[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:13 dist-test-slave-2d0s krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:13 dist-test-slave-2d0s krb5kdc[399](info): set up 2 sockets
Sep 25 13:12:13 dist-test-slave-2d0s krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 13:12:14.451215 392 mini_kdc.cc:121] Time spent starting KDC: real 2.215s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 13:12:14.576081 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.124s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 13:12:14.767998 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.113s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/test-user.keytab.
W20260925 13:12:15.414526 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.646s user 0.001s sys 0.000s
Sep 25 13:12:15 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260925 13:12:15.871794 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.436s user 0.001s sys 0.002s
I20260925 13:12:15.872282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41563
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:15.952221 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:15.952433 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:15.952455 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:15.953534 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:15.953573 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:15.953588 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:15.953600 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:15.953612 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:15.955077 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:15.955282 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:15.955471 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:15.960858 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:15.960971 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:15.960886 412 server_base.cc:1034] running on GCE node
W20260925 13:12:15.960834 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:15.961249 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:15.961563 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:15.962721 412 hybrid_clock.cc:648] HybridClock initialized: now 1790341935962698 us; error 36 us; skew 500 ppm
Sep 25 13:12:15 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:15.965178 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:15.965611 412 webserver.cc:460] Webserver started at http://127.0.98.62:33849/ using document root <none> and password file <none>
I20260925 13:12:15.965860 412 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:15.965929 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:15.966073 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:15.967129 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "b460ca20d07144b79475e9b2dc65dce0"
format_stamp: "Formatted at 2026-09-25 13:12:15 on dist-test-slave-2d0s"
server_key: "ec8c0c0aacf4e63ca67635a54ed8e37e"
server_key_iv: "739424402649d5fd742ca8def07592e5"
server_key_version: "encryptionkey@0"
I20260925 13:12:15.967521 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "b460ca20d07144b79475e9b2dc65dce0"
format_stamp: "Formatted at 2026-09-25 13:12:15 on dist-test-slave-2d0s"
server_key: "ec8c0c0aacf4e63ca67635a54ed8e37e"
server_key_iv: "739424402649d5fd742ca8def07592e5"
server_key_version: "encryptionkey@0"
I20260925 13:12:15.968860 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:15.969602 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:15.969786 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:15.969882 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "b460ca20d07144b79475e9b2dc65dce0"
format_stamp: "Formatted at 2026-09-25 13:12:15 on dist-test-slave-2d0s"
server_key: "ec8c0c0aacf4e63ca67635a54ed8e37e"
server_key_iv: "739424402649d5fd742ca8def07592e5"
server_key_version: "encryptionkey@0"
I20260925 13:12:15.969959 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:15.978991 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:15.981251 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:15.981420 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:15.985266 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41563
I20260925 13:12:15.985335 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41563 every 8 connection(s)
I20260925 13:12:15.985720 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:15.986621 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:15.987087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 412
I20260925 13:12:15.987195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:15.987814 392 external_mini_cluster.cc:1428] Setting key c6a6262086decc168c5c1f8f64f2c954
I20260925 13:12:15.989238 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0: Bootstrap starting.
I20260925 13:12:15.989981 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:15.990309 480 log.cc:824] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:15.991046 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0: No bootstrap required, opened a new log
I20260925 13:12:15.993223 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b460ca20d07144b79475e9b2dc65dce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41563 } }
I20260925 13:12:15.993419 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:15.993455 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b460ca20d07144b79475e9b2dc65dce0, State: Initialized, Role: FOLLOWER
I20260925 13:12:15.993602 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b460ca20d07144b79475e9b2dc65dce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41563 } }
I20260925 13:12:15.993733 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:15.993777 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:15.993829 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:15.994927 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b460ca20d07144b79475e9b2dc65dce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41563 } }
I20260925 13:12:15.995018 480 leader_election.cc:302] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b460ca20d07144b79475e9b2dc65dce0; no voters:
I20260925 13:12:15.995247 480 leader_election.cc:288] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:15.995276 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:15.995389 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [term 1 LEADER]: Becoming Leader. State: Replica: b460ca20d07144b79475e9b2dc65dce0, State: Running, Role: LEADER
I20260925 13:12:15.995548 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:15.995525 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b460ca20d07144b79475e9b2dc65dce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41563 } }
I20260925 13:12:15.995982 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b460ca20d07144b79475e9b2dc65dce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b460ca20d07144b79475e9b2dc65dce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41563 } } }
I20260925 13:12:15.995999 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b460ca20d07144b79475e9b2dc65dce0. Latest consensus state: current_term: 1 leader_uuid: "b460ca20d07144b79475e9b2dc65dce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b460ca20d07144b79475e9b2dc65dce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41563 } } }
I20260925 13:12:15.996172 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:15.996181 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:15.996425 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:15.996991 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 13:12:15 dist-test-slave-2d0s krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:16.003145 494 catalog_manager.cc:1347] Generated new cluster ID: 3bccc99e3d714d46b63cebd1ede821ee
I20260925 13:12:16.003202 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:16.006174 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:15.988627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.98.62:41563)
0925 13:12:15.988934 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:15.988942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:15.989055 (+ 113us) server_negotiation.cc:407] Connection header received
0925 13:12:15.992250 (+ 3195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:15.992257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:15.992309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:15.992417 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:15.994653 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:15.995826 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:15.997444 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:15.997985 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:15.998100 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.001330 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:16.001335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:16.001335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:16.001347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:16.004206 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.004669 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.004672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.004673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.004789 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.005044 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.005045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.005045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.005377 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:16.005535 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:16.005809 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 13:12:16.005950 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":84,"threads_started":1}
I20260925 13:12:16.017614 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:16.018435 494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:16.026727 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b460ca20d07144b79475e9b2dc65dce0: Generated new TSK 0
I20260925 13:12:16.026986 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:16.056278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41563
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:16.165450 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:16.165707 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:16.165750 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:16.166953 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:16.166992 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:16.167018 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:16.168450 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:16.168694 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:16.168977 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:16.173950 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:16.173957 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:16.173938 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:16.174229 506 server_base.cc:1034] running on GCE node
I20260925 13:12:16.174431 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:16.174629 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:16.175792 506 hybrid_clock.cc:648] HybridClock initialized: now 1790341936175776 us; error 30 us; skew 500 ppm
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:16.178817 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:16.179296 506 webserver.cc:460] Webserver started at http://127.0.98.1:44097/ using document root <none> and password file <none>
I20260925 13:12:16.179533 506 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:16.179600 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:16.179733 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:16.180796 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "b31bb74dd624453e9652e04fc15dd72a"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "9e89733fb59af0b55067dcfc39d4fdbf"
server_key_iv: "1bdd5bf2ce3e01a06799f723f4bc4357"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.181176 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b31bb74dd624453e9652e04fc15dd72a"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "9e89733fb59af0b55067dcfc39d4fdbf"
server_key_iv: "1bdd5bf2ce3e01a06799f723f4bc4357"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.182657 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:16.183432 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.183635 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:16.183732 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "b31bb74dd624453e9652e04fc15dd72a"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "9e89733fb59af0b55067dcfc39d4fdbf"
server_key_iv: "1bdd5bf2ce3e01a06799f723f4bc4357"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.183809 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:16.212309 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:16.215085 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:16.215277 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:16.215564 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:16.215888 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:16.215940 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.215976 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:16.216002 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.221419 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43795
I20260925 13:12:16.221491 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43795 every 8 connection(s)
I20260925 13:12:16.221880 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:16.222126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 506
I20260925 13:12:16.222219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:16.222396 392 external_mini_cluster.cc:1428] Setting key b4a359159fb0da9f7a4df6d613fed795
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:12:16.232335 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.222698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57457 (local address 127.0.98.62:41563)
0925 13:12:16.222771 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:16.222773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:16.222995 (+ 222us) server_negotiation.cc:407] Connection header received
0925 13:12:16.223560 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:16.223561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:16.223574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:16.223639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:16.224784 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.225288 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.226347 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.226716 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.226797 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.229149 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:16.229150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:16.229151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:16.229159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:16.230817 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.231275 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.231276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.231277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.231327 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.231655 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.231656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.231656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.231781 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:16.231843 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:16.232179 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 13:12:16.232260 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:12:16.232568 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.222665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41563 (local address 127.0.98.1:57457)
0925 13:12:16.222856 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:16.222903 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:16.223448 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:16.223713 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:16.223716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:16.224390 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:16.224681 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:16.224688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.225373 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.225374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:16.226261 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:16.226263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.226820 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.226821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:16.226938 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.227516 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:16.227524 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:16.229055 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:16.230923 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:16.230929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:16.230932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:16.231199 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:16.231449 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:16.231450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:16.231450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:16.231562 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:16.231873 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:16.231878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:16.232037 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 13:12:16.232152 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 13:12:16.232309 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260925 13:12:16.233119 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41563
I20260925 13:12:16.233224 635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:12:16.233415 635 heartbeater.cc:511] Master 127.0.98.62:41563 requested a full tablet report, sending...
I20260925 13:12:16.234015 444 ts_manager.cc:194] Registered new tserver with Master: b31bb74dd624453e9652e04fc15dd72a (127.0.98.1:43795)
I20260925 13:12:16.234984 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:16.248076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41563
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:16.329139 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:16.329337 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:16.329370 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:16.330559 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:16.330600 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:16.330626 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:16.332151 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:16.332403 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:16.332644 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:16.337582 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:16.337675 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:16.337590 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:16.337832 640 server_base.cc:1034] running on GCE node
I20260925 13:12:16.338018 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:16.338258 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:16.339399 640 hybrid_clock.cc:648] HybridClock initialized: now 1790341936339387 us; error 27 us; skew 500 ppm
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:16.341799 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:16.342306 640 webserver.cc:460] Webserver started at http://127.0.98.2:42835/ using document root <none> and password file <none>
I20260925 13:12:16.342509 640 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:16.342549 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:16.342635 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:16.343621 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "721ea6ab4d3d4555b40bbc447c91ea49"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "5c3f9701f1ff726a01331290c4cc2fe0"
server_key_iv: "e0d0822e2ee541a47e0f42baa55da209"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.343922 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "721ea6ab4d3d4555b40bbc447c91ea49"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "5c3f9701f1ff726a01331290c4cc2fe0"
server_key_iv: "e0d0822e2ee541a47e0f42baa55da209"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.345162 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:16.345894 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.346066 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:16.346167 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "721ea6ab4d3d4555b40bbc447c91ea49"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "5c3f9701f1ff726a01331290c4cc2fe0"
server_key_iv: "e0d0822e2ee541a47e0f42baa55da209"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.346238 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:16.359359 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:16.361318 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:16.361474 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:16.361737 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:16.362023 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:16.362073 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.362112 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:16.362138 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.368058 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39795
I20260925 13:12:16.368098 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39795 every 8 connection(s)
I20260925 13:12:16.368405 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:12:16.372674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 640
I20260925 13:12:16.372754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:16.372969 392 external_mini_cluster.cc:1428] Setting key 7615bd2bdbd558402b1938baeee605ca
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:16.378714 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.369528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41163 (local address 127.0.98.62:41563)
0925 13:12:16.369600 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:16.369603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:16.369745 (+ 142us) server_negotiation.cc:407] Connection header received
0925 13:12:16.370216 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:16.370217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:16.370226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:16.370317 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:16.371276 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.371768 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.372907 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.373206 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.373282 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.375655 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:16.375656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:16.375656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:16.375664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:16.377140 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.377626 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.377627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.377627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.377733 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.378046 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.378047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.378048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.378152 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:16.378199 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:16.378565 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 13:12:16.378631 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:12:16.379014 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.369455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41563 (local address 127.0.98.2:41163)
0925 13:12:16.369596 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:16.369632 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:16.370138 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:16.370397 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:16.370400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:16.370988 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:16.371197 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:16.371201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.371851 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.371852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:16.372764 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:16.372767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.373287 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.373288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:16.373440 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.374080 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:16.374090 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:16.375561 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:16.377217 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:16.377224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:16.377226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:16.377550 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:16.377807 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:16.377808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:16.377809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:16.377971 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:16.378215 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:16.378221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:16.378402 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 13:12:16.378526 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 13:12:16.378741 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260925 13:12:16.379526 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41563
I20260925 13:12:16.379639 769 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:16.379835 769 heartbeater.cc:511] Master 127.0.98.62:41563 requested a full tablet report, sending...
I20260925 13:12:16.380249 444 ts_manager.cc:194] Registered new tserver with Master: 721ea6ab4d3d4555b40bbc447c91ea49 (127.0.98.2:39795)
I20260925 13:12:16.380641 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41163
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:16.396958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41563
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:16.478896 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:16.479110 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:16.479133 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:16.480269 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:16.480311 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:16.480365 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:16.481829 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:16.482059 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:16.482299 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:16.487150 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:16.487174 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:16.487391 774 server_base.cc:1034] running on GCE node
W20260925 13:12:16.487425 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:16.487679 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:16.487923 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:16.489082 774 hybrid_clock.cc:648] HybridClock initialized: now 1790341936489071 us; error 27 us; skew 500 ppm
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:16.491518 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:16.491928 774 webserver.cc:460] Webserver started at http://127.0.98.3:44183/ using document root <none> and password file <none>
I20260925 13:12:16.492152 774 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:16.492218 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:16.492365 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:16.493357 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "81f2cf98a3b0435f80283bec3b809b91"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "8f66caf06f12e5c916ef8f27e701fb29"
server_key_iv: "81304094f8928ad4b75b1aaefb5258b2"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.493758 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81f2cf98a3b0435f80283bec3b809b91"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "8f66caf06f12e5c916ef8f27e701fb29"
server_key_iv: "81304094f8928ad4b75b1aaefb5258b2"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.495072 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:16.495900 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.496063 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:16.496161 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "81f2cf98a3b0435f80283bec3b809b91"
format_stamp: "Formatted at 2026-09-25 13:12:16 on dist-test-slave-2d0s"
server_key: "8f66caf06f12e5c916ef8f27e701fb29"
server_key_iv: "81304094f8928ad4b75b1aaefb5258b2"
server_key_version: "encryptionkey@0"
I20260925 13:12:16.496234 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:16.503103 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:16.505120 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:16.505286 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:16.505486 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:16.505822 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:16.505882 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.505923 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:16.505949 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:16.512213 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34439
I20260925 13:12:16.512292 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34439 every 8 connection(s)
I20260925 13:12:16.512598 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:16.521543 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.513359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57411 (local address 127.0.98.62:41563)
0925 13:12:16.513406 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:16.513408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:16.513575 (+ 167us) server_negotiation.cc:407] Connection header received
0925 13:12:16.514119 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:16.514121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:16.514132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:16.514196 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:16.515073 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.515597 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.516620 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.516822 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.516886 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.519131 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:16.519133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:16.519134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:16.519140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:16.520320 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.520778 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.520779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.520779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.520836 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.521042 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.521043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.521043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.521138 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:16.521186 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:16.521406 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 13:12:16.521463 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:16.521731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 774
I20260925 13:12:16.521796 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.513282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41563 (local address 127.0.98.3:57411)
0925 13:12:16.513420 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:16.513459 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:16.514011 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:16.514249 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:16.514252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:16.514765 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:16.514954 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:16.514958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.515702 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.515703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:16.516536 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:16.516538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.516899 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.516900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:16.516996 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.517459 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:16.517467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:16.519027 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:16.520414 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:16.520419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:16.520421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:16.520705 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:16.520894 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:16.520895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:16.520895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:16.520989 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:16.521190 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:16.521193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:16.521298 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 13:12:16.521378 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 13:12:16.521491 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260925 13:12:16.521832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:16.521994 392 external_mini_cluster.cc:1428] Setting key a54ce0da4538cfe33cc5a50dcd2bd103
I20260925 13:12:16.522464 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41563
I20260925 13:12:16.522574 903 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:16.522746 903 heartbeater.cc:511] Master 127.0.98.62:41563 requested a full tablet report, sending...
I20260925 13:12:16.523097 443 ts_manager.cc:194] Registered new tserver with Master: 81f2cf98a3b0435f80283bec3b809b91 (127.0.98.3:34439)
I20260925 13:12:16.523528 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57411
I20260925 13:12:16.534231 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:16.556455 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.548812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.0.98.1:43795)
0925 13:12:16.549010 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:16.549016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:16.549028 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:12:16.549065 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:16.549067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:16.549083 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:16.549174 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:16.549873 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.550634 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.551372 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.551696 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.551754 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.553936 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:16.553943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:16.553943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:16.553950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:16.555531 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.555846 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.555848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.555849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.555902 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.556048 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.556048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.556048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.556163 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:16.556259 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:16.556320 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 13:12:16.556378 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
W20260925 13:12:16.556846 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44580
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:16.578971 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:16.571302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.98.1:43795)
0925 13:12:16.571373 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:16.571375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:16.571441 (+ 66us) server_negotiation.cc:407] Connection header received
0925 13:12:16.571511 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:16.571512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:16.571524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:16.571584 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:16.572223 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.572772 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.573542 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:16.573850 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:16.573901 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:16.576318 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:16.576320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:16.576320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:16.576329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:16.577846 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.578193 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.578194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.578195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.578250 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:16.578454 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:16.578455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:16.578456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:16.578554 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:16.578694 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:16.578779 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 13:12:16.578865 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 13:12:16.579573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 506
I20260925 13:12:16.584213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 640
I20260925 13:12:16.589193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 774
I20260925 13:12:16.593204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 412
2026-09-25T13:12:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4384 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[917](info): set up 2 sockets
Sep 25 13:12:16 dist-test-slave-2d0s krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 13:12:17.625252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:17 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:17.737401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:17.814208 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:17.814396 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:17.814416 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:17.815481 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:17.815514 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:17.815528 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:17.815541 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:17.815552 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:17.816953 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:17.817127 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:17.817288 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:17.821810 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:17.821825 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:17.821815 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:17.822099 930 server_base.cc:1034] running on GCE node
I20260925 13:12:17.822389 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:17.822696 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:17.823854 930 hybrid_clock.cc:648] HybridClock initialized: now 1790341937823842 us; error 23 us; skew 500 ppm
Sep 25 13:12:17 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:17.830778 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:17.831286 930 webserver.cc:460] Webserver started at http://127.0.98.62:44299/ using document root <none> and password file <none>
I20260925 13:12:17.831537 930 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:17.831609 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:17.831763 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:17.832831 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "0166072368084345a20094e8bff4d172"
format_stamp: "Formatted at 2026-09-25 13:12:17 on dist-test-slave-2d0s"
server_key: "a8c5652f10618931d2545085ed9e7e5e"
server_key_iv: "d07c4b34e16ed71523d61ac7bc2500a0"
server_key_version: "encryptionkey@0"
I20260925 13:12:17.833214 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "0166072368084345a20094e8bff4d172"
format_stamp: "Formatted at 2026-09-25 13:12:17 on dist-test-slave-2d0s"
server_key: "a8c5652f10618931d2545085ed9e7e5e"
server_key_iv: "d07c4b34e16ed71523d61ac7bc2500a0"
server_key_version: "encryptionkey@0"
I20260925 13:12:17.834663 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:17.835438 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:17.835628 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:17.835745 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "0166072368084345a20094e8bff4d172"
format_stamp: "Formatted at 2026-09-25 13:12:17 on dist-test-slave-2d0s"
server_key: "a8c5652f10618931d2545085ed9e7e5e"
server_key_iv: "d07c4b34e16ed71523d61ac7bc2500a0"
server_key_version: "encryptionkey@0"
I20260925 13:12:17.835825 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:17.851217 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:17.852926 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:17.853101 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:17.857013 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42447
I20260925 13:12:17.857046 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42447 every 8 connection(s)
I20260925 13:12:17.857432 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:17.858260 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:17.860325 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172: Bootstrap starting.
I20260925 13:12:17.860852 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:17.861084 998 log.cc:824] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:17.861780 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172: No bootstrap required, opened a new log
I20260925 13:12:17.861838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 930
I20260925 13:12:17.861927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:17.862092 392 external_mini_cluster.cc:1428] Setting key 82ef4f053a4ba31bf87e7aafc7b45474
I20260925 13:12:17.864483 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166072368084345a20094e8bff4d172" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260925 13:12:17.864641 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:17.864689 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0166072368084345a20094e8bff4d172, State: Initialized, Role: FOLLOWER
I20260925 13:12:17.864859 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166072368084345a20094e8bff4d172" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260925 13:12:17.864957 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:17.865005 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:17.865054 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:17.865749 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166072368084345a20094e8bff4d172" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260925 13:12:17.865882 998 leader_election.cc:302] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0166072368084345a20094e8bff4d172; no voters:
I20260925 13:12:17.866057 998 leader_election.cc:288] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:17.866118 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:17.866271 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [term 1 LEADER]: Becoming Leader. State: Replica: 0166072368084345a20094e8bff4d172, State: Running, Role: LEADER
I20260925 13:12:17.866331 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:17.866356 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166072368084345a20094e8bff4d172" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260925 13:12:17.866670 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0166072368084345a20094e8bff4d172. Latest consensus state: current_term: 1 leader_uuid: "0166072368084345a20094e8bff4d172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166072368084345a20094e8bff4d172" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } } }
I20260925 13:12:17.866751 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:17.866679 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0166072368084345a20094e8bff4d172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166072368084345a20094e8bff4d172" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } } }
I20260925 13:12:17.866871 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172 [sys.catalog]: This master's current role is: LEADER
W20260925 13:12:17.868013 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:12:17.868080 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:17.868223 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:17.868462 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:17.870040 1009 catalog_manager.cc:1347] Generated new cluster ID: c461de6e48034d498cef53c75a9ea76e
I20260925 13:12:17.870093 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:17 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:17.873960 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:17.862735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.0.98.62:42447)
0925 13:12:17.862906 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:17.862919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:17.862952 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:12:17.863393 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:17.863400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:17.863458 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:17.863572 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:17.864403 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:17.865762 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:17.866915 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:17.867534 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:17.867625 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:17.870777 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:17.870787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:17.870788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:17.870799 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:17.872511 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:17.872845 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:17.872848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:17.872848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:17.872947 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:17.873117 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:17.873118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:17.873118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:17.873390 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:17.873525 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:17.873587 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 13:12:17.873758 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260925 13:12:17.886545 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:17.887220 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:17.896461 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0166072368084345a20094e8bff4d172: Generated new TSK 0
I20260925 13:12:17.896646 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:17.920629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:18.000675 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:18.000908 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:18.000934 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:18.002148 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:18.002193 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:18.002254 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:18.003715 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:18.003968 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:18.004215 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:18.009234 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:18.009240 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:18.009493 1024 server_base.cc:1034] running on GCE node
W20260925 13:12:18.009297 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:18.009799 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:18.010056 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:18.011215 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790341938011196 us; error 40 us; skew 500 ppm
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:18.013855 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:18.014238 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42639/ using document root <none> and password file <none>
I20260925 13:12:18.014410 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:18.014449 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:18.014533 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:18.015869 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec61d49972ec451cac7a75ad754516e2"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "15e983463d32a0b272dc1d896c1ede54"
server_key_iv: "a3a1e64e44e225ecca3b9f81a5635652"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.016357 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec61d49972ec451cac7a75ad754516e2"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "15e983463d32a0b272dc1d896c1ede54"
server_key_iv: "a3a1e64e44e225ecca3b9f81a5635652"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.017975 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 13:12:18.018863 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.019027 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:18.019136 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "ec61d49972ec451cac7a75ad754516e2"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "15e983463d32a0b272dc1d896c1ede54"
server_key_iv: "a3a1e64e44e225ecca3b9f81a5635652"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.019208 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:18.055763 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:18.057811 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:18.057965 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:18.058229 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:18.058563 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:18.058598 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.058641 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:18.058669 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.064738 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38029
I20260925 13:12:18.064795 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38029 every 8 connection(s)
I20260925 13:12:18.065171 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:18.065478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1024
I20260925 13:12:18.065577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:18.065788 392 external_mini_cluster.cc:1428] Setting key 3fc3a96c17188a9858f637a34634f47e
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:12:18.076251 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.066248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58623 (local address 127.0.98.62:42447)
0925 13:12:18.066289 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.066291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.066300 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:18.066836 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.066838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.066849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.066898 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.068261 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.069123 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.070748 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.070953 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.071009 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.073402 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.073403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.073403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.073426 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.074918 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.075332 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.075333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.075334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.075423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.075703 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.075704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.075704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.075816 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.075871 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.076110 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.076182 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:18.076403 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.065976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42447 (local address 127.0.98.1:58623)
0925 13:12:18.066165 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.066207 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.066726 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.067007 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.067012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.067889 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:18.068173 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.068178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.069212 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.069213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.070657 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.070660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.071056 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.071058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.071226 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.071755 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:18.071767 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:18.073307 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:18.075009 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:18.075015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:18.075018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:18.075242 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:18.075492 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:18.075493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:18.075494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:18.075624 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:18.075876 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:18.075881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:18.076003 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.076086 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.076205 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":117,"threads_started":1}
I20260925 13:12:18.076960 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42447
I20260925 13:12:18.077064 1153 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:12:18.077271 1153 heartbeater.cc:511] Master 127.0.98.62:42447 requested a full tablet report, sending...
I20260925 13:12:18.077839 962 ts_manager.cc:194] Registered new tserver with Master: ec61d49972ec451cac7a75ad754516e2 (127.0.98.1:38029)
I20260925 13:12:18.078464 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:18.090710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:18.171818 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:18.172060 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:18.172098 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:18.173360 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:18.173432 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:18.173480 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:18.174973 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:18.175236 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:18.175462 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:18.180219 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:18.180249 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:18.180546 1158 server_base.cc:1034] running on GCE node
W20260925 13:12:18.180220 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:18.180785 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:18.181015 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:18.182168 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790341938182167 us; error 36 us; skew 500 ppm
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:18.184437 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:18.184897 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42987/ using document root <none> and password file <none>
I20260925 13:12:18.185118 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:18.185180 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:18.185317 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:18.186439 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "99b2d88c8f994a0fbc1a800be9ff7f53"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "8ad629a0a9efe2df47c36a98526b95aa"
server_key_iv: "010d82ce330c356358710e6a0b78d657"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.186805 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "99b2d88c8f994a0fbc1a800be9ff7f53"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "8ad629a0a9efe2df47c36a98526b95aa"
server_key_iv: "010d82ce330c356358710e6a0b78d657"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.188094 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:18.188756 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.188939 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:18.189039 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "99b2d88c8f994a0fbc1a800be9ff7f53"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "8ad629a0a9efe2df47c36a98526b95aa"
server_key_iv: "010d82ce330c356358710e6a0b78d657"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.189116 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:18.199601 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:18.201388 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:18.201537 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:18.201768 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:18.202080 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:18.202135 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.202175 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:18.202200 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.207852 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40583
I20260925 13:12:18.207937 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40583 every 8 connection(s)
I20260925 13:12:18.208241 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:18.215301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1158
I20260925 13:12:18.215384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:18.215580 392 external_mini_cluster.cc:1428] Setting key a0fc038a83c5c8f56de940b27841bf80
I20260925 13:12:18.218011 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.208993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46011 (local address 127.0.98.62:42447)
0925 13:12:18.209057 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.209060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.209267 (+ 207us) server_negotiation.cc:407] Connection header received
0925 13:12:18.209873 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.209874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.209883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.209925 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.210872 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.211374 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.212347 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.212529 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.212578 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.214716 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.214717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.214718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.214725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.216441 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.216919 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.216920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.216920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.216979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.217238 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.217239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.217239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.217350 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.217397 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.217787 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.217892 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:12:18.218149 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.208963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42447 (local address 127.0.98.2:46011)
0925 13:12:18.209139 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.209181 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.209754 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.209984 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.209988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.210596 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:18.210793 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.210799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.211470 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.211471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.212271 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.212273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.212618 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.212619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.212752 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.213199 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:18.213207 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:18.214632 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:18.216526 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:18.216532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:18.216534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:18.216843 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:18.217040 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:18.217041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:18.217041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:18.217179 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:18.217409 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:18.217414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:18.217601 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.217756 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.217941 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260925 13:12:18.218760 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42447
I20260925 13:12:18.218858 1287 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:18.219038 1287 heartbeater.cc:511] Master 127.0.98.62:42447 requested a full tablet report, sending...
I20260925 13:12:18.219408 962 ts_manager.cc:194] Registered new tserver with Master: 99b2d88c8f994a0fbc1a800be9ff7f53 (127.0.98.2:40583)
I20260925 13:12:18.219949 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46011
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:18.241456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:18.321000 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:18.321197 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:18.321220 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:18.322362 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:18.322397 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:18.322423 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:18.323835 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:18.324057 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:18.324241 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:18.329419 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:18.329424 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:18.329491 1292 server_base.cc:1034] running on GCE node
W20260925 13:12:18.329474 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:18.329878 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:18.330139 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:18.331300 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790341938331287 us; error 45 us; skew 500 ppm
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:18.333633 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:18.334137 1292 webserver.cc:460] Webserver started at http://127.0.98.3:44109/ using document root <none> and password file <none>
I20260925 13:12:18.334369 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:18.334435 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:18.334589 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:18.335660 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d623910b24f4102b698165395ec4fcc"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "1c18b00852aabd662759c0954381d9d7"
server_key_iv: "8bd5a2dc6a3877e727808a6bc3b44f3e"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.336037 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d623910b24f4102b698165395ec4fcc"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "1c18b00852aabd662759c0954381d9d7"
server_key_iv: "8bd5a2dc6a3877e727808a6bc3b44f3e"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.337347 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:18.338093 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.338276 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.338357 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "1d623910b24f4102b698165395ec4fcc"
format_stamp: "Formatted at 2026-09-25 13:12:18 on dist-test-slave-2d0s"
server_key: "1c18b00852aabd662759c0954381d9d7"
server_key_iv: "8bd5a2dc6a3877e727808a6bc3b44f3e"
server_key_version: "encryptionkey@0"
I20260925 13:12:18.338406 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:18.369781 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:18.371755 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:18.371865 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:18.372052 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:18.372391 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:18.372424 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.372466 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:18.372520 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:18.378260 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37829
I20260925 13:12:18.378300 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37829 every 8 connection(s)
I20260925 13:12:18.378644 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:18.386130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1292
I20260925 13:12:18.386238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:18.386402 392 external_mini_cluster.cc:1428] Setting key 36329a227880974c0d73eabf69abf3fd
I20260925 13:12:18.388118 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.379328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57699 (local address 127.0.98.62:42447)
0925 13:12:18.379398 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.379399 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.379622 (+ 223us) server_negotiation.cc:407] Connection header received
0925 13:12:18.380116 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.380117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.380127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.380189 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.381113 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.381657 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.382677 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.382861 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.382911 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.384930 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.384931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.384932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.384938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.386647 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.387122 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.387123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.387123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.387174 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.387478 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.387479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.387479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.387605 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.387681 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.387947 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.388026 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:12:18.388324 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.379300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42447 (local address 127.0.98.3:57699)
0925 13:12:18.379489 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.379532 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.380032 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.380263 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.380266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.380823 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:18.381014 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.381019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.381786 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.381786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.382591 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.382592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.382927 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.382928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.383024 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.383494 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:18.383502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:18.384850 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:18.386758 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:18.386766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:18.386769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:18.387029 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:18.387240 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:18.387241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:18.387241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:18.387357 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:18.387672 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:18.387677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:18.387810 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.387927 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.388086 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260925 13:12:18.388749 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42447
I20260925 13:12:18.388857 1421 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:18.389038 1421 heartbeater.cc:511] Master 127.0.98.62:42447 requested a full tablet report, sending...
I20260925 13:12:18.389416 962 ts_manager.cc:194] Registered new tserver with Master: 1d623910b24f4102b698165395ec4fcc (127.0.98.3:37829)
I20260925 13:12:18.389833 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57699
I20260925 13:12:18.398344 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:12:18.406924 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.399953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.98.62:42447)
0925 13:12:18.400017 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.400019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.400168 (+ 149us) server_negotiation.cc:407] Connection header received
0925 13:12:18.400260 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.400261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.400278 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.400344 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.401137 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.401774 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.402838 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.403153 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.403212 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.404534 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.404535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.404536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.404543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.405737 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.406086 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.406087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.406088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.406143 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.406361 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.406362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.406362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.406513 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.406618 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.406733 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.406820 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:18.409216 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:18.411386 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:18.418555 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.413962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.98.2:40583)
0925 13:12:18.414386 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.414390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.414399 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:18.414443 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.414444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.414476 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.414568 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:18.414736 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.415921 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.417481 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.418195 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.418263 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.418318 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.418434 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.418476 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":77,"threads_started":1}
I20260925 13:12:18.418560 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.413828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40583 (local address 127.0.0.1:34030)
0925 13:12:18.413990 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.414033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.414098 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.414572 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.414576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.414578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:18.414651 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.414653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.416041 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.416042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.417397 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.417399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.418282 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.418283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.418358 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.418362 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.418410 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.418454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":76,"thread_start_us":56,"threads_started":1}
I20260925 13:12:18.418824 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.413834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38029 (local address 127.0.0.1:44770)
0925 13:12:18.414657 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.414667 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.414775 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.415063 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.415064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.415067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:18.415118 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.415120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.415988 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.415990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.417396 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.417399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.418435 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.418436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.418489 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.418493 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.418537 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.418570 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":93,"thread_start_us":89,"threads_started":1}
I20260925 13:12:18.418944 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.414031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37829 (local address 127.0.0.1:58066)
0925 13:12:18.414871 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.414879 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.414922 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.415098 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.415099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.415101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:18.415140 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.415141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.416326 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.416327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.417193 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.417195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.418199 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.418200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.418279 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.418293 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.418357 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.418414 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":76,"threads_started":1}
I20260925 13:12:18.419060 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.414158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.0.98.1:38029)
0925 13:12:18.414337 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.414342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.414749 (+ 407us) server_negotiation.cc:407] Connection header received
0925 13:12:18.414873 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.414876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.414888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.414979 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:18.415186 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.415886 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.417737 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.418276 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.418345 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.418403 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.418759 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.418811 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260925 13:12:18.418725 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.414129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.0.98.3:37829)
0925 13:12:18.414297 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.414302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.414945 (+ 643us) server_negotiation.cc:407] Connection header received
0925 13:12:18.414988 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.414990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.415005 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.415083 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:18.415221 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.416230 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.417293 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.418036 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.418107 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.418163 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.418497 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.418536 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260925 13:12:18.419480 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 09c3214edb70407faf6d0f61ee5cbf7a (DEFAULT_TABLE table=test-table [id=fde5f9cc798c4340b7aab3be769b3ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:18.419669 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 09c3214edb70407faf6d0f61ee5cbf7a (DEFAULT_TABLE table=test-table [id=fde5f9cc798c4340b7aab3be769b3ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:18.419845 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c3214edb70407faf6d0f61ee5cbf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:18.419919 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c3214edb70407faf6d0f61ee5cbf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:18.419842 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 09c3214edb70407faf6d0f61ee5cbf7a (DEFAULT_TABLE table=test-table [id=fde5f9cc798c4340b7aab3be769b3ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:18.420143 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c3214edb70407faf6d0f61ee5cbf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:18.422580 1438 tablet_bootstrap.cc:492] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc: Bootstrap starting.
I20260925 13:12:18.423223 1438 tablet_bootstrap.cc:654] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:18.423413 1440 tablet_bootstrap.cc:492] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: Bootstrap starting.
I20260925 13:12:18.424054 1440 tablet_bootstrap.cc:654] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:18.424048 1439 tablet_bootstrap.cc:492] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: Bootstrap starting.
I20260925 13:12:18.424244 1438 log.cc:824] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:18.424620 1440 log.cc:824] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:18.424662 1439 tablet_bootstrap.cc:654] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:18.425155 1439 log.cc:824] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:18.425488 1438 tablet_bootstrap.cc:492] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc: No bootstrap required, opened a new log
I20260925 13:12:18.425544 1438 ts_tablet_manager.cc:1397] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:12:18.425945 1439 tablet_bootstrap.cc:492] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: No bootstrap required, opened a new log
I20260925 13:12:18.426011 1439 ts_tablet_manager.cc:1397] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:18.426206 1440 tablet_bootstrap.cc:492] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: No bootstrap required, opened a new log
I20260925 13:12:18.426285 1440 ts_tablet_manager.cc:1397] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 13:12:18.427477 1438 raft_consensus.cc:348] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.427547 1440 raft_consensus.cc:348] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.427662 1438 raft_consensus.cc:374] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:18.427685 1440 raft_consensus.cc:374] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:18.427696 1438 raft_consensus.cc:729] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d623910b24f4102b698165395ec4fcc, State: Initialized, Role: FOLLOWER
I20260925 13:12:18.427728 1440 raft_consensus.cc:729] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec61d49972ec451cac7a75ad754516e2, State: Initialized, Role: FOLLOWER
I20260925 13:12:18.427827 1440 consensus_queue.cc:260] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.427812 1438 consensus_queue.cc:260] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.427862 1439 raft_consensus.cc:348] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.428018 1439 raft_consensus.cc:374] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:18.428061 1439 raft_consensus.cc:729] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b2d88c8f994a0fbc1a800be9ff7f53, State: Initialized, Role: FOLLOWER
I20260925 13:12:18.428114 1438 ts_tablet_manager.cc:1428] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:18.428114 1440 ts_tablet_manager.cc:1428] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:12:18.428223 1421 heartbeater.cc:503] Master 127.0.98.62:42447 was elected leader, sending a full tablet report...
I20260925 13:12:18.428200 1439 consensus_queue.cc:260] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.428260 1153 heartbeater.cc:503] Master 127.0.98.62:42447 was elected leader, sending a full tablet report...
I20260925 13:12:18.428464 1287 heartbeater.cc:503] Master 127.0.98.62:42447 was elected leader, sending a full tablet report...
I20260925 13:12:18.428539 1439 ts_tablet_manager.cc:1428] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260925 13:12:18.458832 1288 tablet.cc:2369] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:18.542857 1444 raft_consensus.cc:482] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:18.543051 1444 raft_consensus.cc:504] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.543474 1444 leader_election.cc:288] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99b2d88c8f994a0fbc1a800be9ff7f53 (127.0.98.2:40583), 1d623910b24f4102b698165395ec4fcc (127.0.98.3:37829)
I20260925 13:12:18.546633 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.543848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45969 (local address 127.0.98.2:40583)
0925 13:12:18.543931 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.543933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.543941 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:18.543971 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.543972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.543985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.544048 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:18.544221 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.544745 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.545883 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.546241 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.546296 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.546303 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.546552 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.546581 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 13:12:18.546648 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.543682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40583 (local address 127.0.98.1:45969)
0925 13:12:18.543761 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.543774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.543838 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.544067 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.544069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.544071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:18.544148 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.544150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.544845 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.544845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.545769 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.545772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.546397 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.546398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.546476 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.546480 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.546537 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.546574 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":37}
I20260925 13:12:18.547011 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c3214edb70407faf6d0f61ee5cbf7a" candidate_uuid: "ec61d49972ec451cac7a75ad754516e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" is_pre_election: true
I20260925 13:12:18.547178 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.543805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37829 (local address 127.0.98.1:51037)
0925 13:12:18.543954 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:18.543963 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:18.544014 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:18.544204 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:18.544205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:18.544207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:18.544278 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.544280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.544932 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.544933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.545765 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:18.545767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.546910 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.546911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:18.546980 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.546983 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:18.547060 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 13:12:18.547107 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":9,"thread_start_us":88,"threads_started":1}
I20260925 13:12:18.547169 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.543898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51037 (local address 127.0.98.3:37829)
0925 13:12:18.543946 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.543948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.544069 (+ 121us) server_negotiation.cc:407] Connection header received
0925 13:12:18.544094 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.544095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.544107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.544165 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:18.544374 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.544856 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.545848 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.546842 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.546904 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.546913 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.547064 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.547112 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:18.547204 1241 raft_consensus.cc:2454] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec61d49972ec451cac7a75ad754516e2 in term 0.
I20260925 13:12:18.547434 1042 leader_election.cc:302] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99b2d88c8f994a0fbc1a800be9ff7f53, ec61d49972ec451cac7a75ad754516e2; no voters:
I20260925 13:12:18.547477 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c3214edb70407faf6d0f61ee5cbf7a" candidate_uuid: "ec61d49972ec451cac7a75ad754516e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d623910b24f4102b698165395ec4fcc" is_pre_election: true
I20260925 13:12:18.547566 1444 raft_consensus.cc:2781] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:18.547613 1444 raft_consensus.cc:482] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:18.547611 1375 raft_consensus.cc:2454] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec61d49972ec451cac7a75ad754516e2 in term 0.
I20260925 13:12:18.547640 1444 raft_consensus.cc:3037] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:18.548327 1444 raft_consensus.cc:504] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.548446 1444 leader_election.cc:288] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [CANDIDATE]: Term 1 election: Requested vote from peers 99b2d88c8f994a0fbc1a800be9ff7f53 (127.0.98.2:40583), 1d623910b24f4102b698165395ec4fcc (127.0.98.3:37829)
I20260925 13:12:18.548616 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c3214edb70407faf6d0f61ee5cbf7a" candidate_uuid: "ec61d49972ec451cac7a75ad754516e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53"
I20260925 13:12:18.548640 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c3214edb70407faf6d0f61ee5cbf7a" candidate_uuid: "ec61d49972ec451cac7a75ad754516e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d623910b24f4102b698165395ec4fcc"
I20260925 13:12:18.548710 1241 raft_consensus.cc:3037] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:18.548708 1375 raft_consensus.cc:3037] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:18.549403 1375 raft_consensus.cc:2454] T 09c3214edb70407faf6d0f61ee5cbf7a P 1d623910b24f4102b698165395ec4fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec61d49972ec451cac7a75ad754516e2 in term 1.
I20260925 13:12:18.549539 1241 raft_consensus.cc:2454] T 09c3214edb70407faf6d0f61ee5cbf7a P 99b2d88c8f994a0fbc1a800be9ff7f53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec61d49972ec451cac7a75ad754516e2 in term 1.
I20260925 13:12:18.549579 1042 leader_election.cc:302] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d623910b24f4102b698165395ec4fcc, ec61d49972ec451cac7a75ad754516e2; no voters:
I20260925 13:12:18.549712 1444 raft_consensus.cc:2781] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:18.549892 1444 raft_consensus.cc:686] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [term 1 LEADER]: Becoming Leader. State: Replica: ec61d49972ec451cac7a75ad754516e2, State: Running, Role: LEADER
I20260925 13:12:18.550014 1444 consensus_queue.cc:237] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } }
I20260925 13:12:18.550904 962 catalog_manager.cc:5495] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec61d49972ec451cac7a75ad754516e2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ec61d49972ec451cac7a75ad754516e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b2d88c8f994a0fbc1a800be9ff7f53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d623910b24f4102b698165395ec4fcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec61d49972ec451cac7a75ad754516e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38029 } health_report { overall_health: HEALTHY } } }
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:18.565145 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.558454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.0.98.1:38029)
0925 13:12:18.558524 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.558526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.558583 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:12:18.558639 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.558640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.558649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.558686 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.559190 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.559642 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.560293 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.560462 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.560508 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.562801 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.562807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.562808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.562821 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.564215 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.564532 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.564534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.564535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.564582 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.564735 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.564735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.564736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.564855 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.564958 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.565005 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.565060 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 13:12:18.565914 1154 tablet.cc:2369] T 09c3214edb70407faf6d0f61ee5cbf7a P ec61d49972ec451cac7a75ad754516e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:18.587332 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.580800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.0.98.1:38029)
0925 13:12:18.580894 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.580896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.580953 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:12:18.580997 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.580998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.581006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.581053 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.581541 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.582066 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.582764 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.582957 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.583008 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.585055 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.585056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.585057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.585062 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.586349 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.586660 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.586660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.586661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.586707 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.586871 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.586871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.586871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.586952 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.587025 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.587167 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.587221 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:18.609052 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:18.602240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.1:38029)
0925 13:12:18.602312 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:18.602315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:18.602417 (+ 102us) server_negotiation.cc:407] Connection header received
0925 13:12:18.602487 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:18.602488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:18.602498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:18.602566 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:18.603104 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.603568 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.604301 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:18.604493 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:18.604571 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:18.606804 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:18.606805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:18.606805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:18.606812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:18.607961 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.608295 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.608296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.608296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.608350 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:18.608564 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:18.608565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:18.608566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:18.608663 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:18.608762 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:18.608867 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 13:12:18.608948 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 13:12:18.609526 1067 tablet_service.cc:2908] Rejecting scan request for tablet 09c3214edb70407faf6d0f61ee5cbf7a: Uninitialized: clean time has not yet been initialized
I20260925 13:12:18.610361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1024
I20260925 13:12:18.615195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1158
I20260925 13:12:18.619640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1292
I20260925 13:12:18.623591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 930
2026-09-25T13:12:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2030 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[1462](info): set up 2 sockets
Sep 25 13:12:18 dist-test-slave-2d0s krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260925 13:12:19.653173 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:19 dist-test-slave-2d0s krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:19.764467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:19.842697 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:19.842912 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:19.842952 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:19.844055 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:19.844112 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 13:12:19.844134 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:19.844156 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:19.844182 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:19.844206 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:19.845688 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36463
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:19.845932 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:19.846143 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:19.850525 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:19.850531 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:19.850508 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:19.850756 1475 server_base.cc:1034] running on GCE node
I20260925 13:12:19.850978 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:19.851264 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:19.852417 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790341939852400 us; error 33 us; skew 500 ppm
Sep 25 13:12:19 dist-test-slave-2d0s krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:19.854941 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:19.855388 1475 webserver.cc:460] Webserver started at http://127.0.98.62:43297/ using document root <none> and password file <none>
I20260925 13:12:19.855626 1475 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:19.855692 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:19.855835 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:19.856801 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "64a705a833884b869aecac965319f6f2"
format_stamp: "Formatted at 2026-09-25 13:12:19 on dist-test-slave-2d0s"
server_key: "c8d807bee27640d3aa393d6cf6214672"
server_key_iv: "f208abcda40a12dcc4abb6c3e89d16ff"
server_key_version: "encryptionkey@0"
I20260925 13:12:19.857144 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "64a705a833884b869aecac965319f6f2"
format_stamp: "Formatted at 2026-09-25 13:12:19 on dist-test-slave-2d0s"
server_key: "c8d807bee27640d3aa393d6cf6214672"
server_key_iv: "f208abcda40a12dcc4abb6c3e89d16ff"
server_key_version: "encryptionkey@0"
I20260925 13:12:19.858475 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:19.859133 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:19.859305 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:19.859411 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "64a705a833884b869aecac965319f6f2"
format_stamp: "Formatted at 2026-09-25 13:12:19 on dist-test-slave-2d0s"
server_key: "c8d807bee27640d3aa393d6cf6214672"
server_key_iv: "f208abcda40a12dcc4abb6c3e89d16ff"
server_key_version: "encryptionkey@0"
I20260925 13:12:19.859479 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:19.876763 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:19.879297 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:19.879455 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:19.883595 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36463
I20260925 13:12:19.883673 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36463 every 8 connection(s)
I20260925 13:12:19.883958 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:19.884552 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:19.886605 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: Bootstrap starting.
I20260925 13:12:19.887187 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:19.887454 1553 log.cc:824] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:19.888024 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: No bootstrap required, opened a new log
I20260925 13:12:19.888360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1475
I20260925 13:12:19.888460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:19.888597 392 external_mini_cluster.cc:1428] Setting key e2f22d94c85c6af980131746dc0b6c58
I20260925 13:12:19.890463 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a705a833884b869aecac965319f6f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260925 13:12:19.890571 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:19.890590 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a705a833884b869aecac965319f6f2, State: Initialized, Role: FOLLOWER
I20260925 13:12:19.890707 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a705a833884b869aecac965319f6f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260925 13:12:19.890772 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:19.890792 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:19.890892 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:19.891444 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a705a833884b869aecac965319f6f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260925 13:12:19.891556 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a705a833884b869aecac965319f6f2; no voters:
I20260925 13:12:19.891692 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:19.891719 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:19.891886 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 64a705a833884b869aecac965319f6f2, State: Running, Role: LEADER
I20260925 13:12:19.891978 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:19.892015 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a705a833884b869aecac965319f6f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260925 13:12:19.892431 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a705a833884b869aecac965319f6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a705a833884b869aecac965319f6f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } } }
I20260925 13:12:19.892518 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:19.892670 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a705a833884b869aecac965319f6f2. Latest consensus state: current_term: 1 leader_uuid: "64a705a833884b869aecac965319f6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a705a833884b869aecac965319f6f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } } }
I20260925 13:12:19.892728 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:19.892944 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:19.893452 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:19.895107 1566 catalog_manager.cc:1347] Generated new cluster ID: 23ffd755236041719b5505ec673a6815
I20260925 13:12:19.895160 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:19 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:19.900279 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:19.889022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.0.98.62:36463)
0925 13:12:19.889182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:19.889190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:19.889216 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:12:19.890089 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:19.890097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:19.890150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:19.890286 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:19.890881 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:19.891978 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:19.893049 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:19.893654 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:19.894081 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:19.896747 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:19.896751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:19.896751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:19.896762 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:19.898762 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:19.899119 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:19.899122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:19.899122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:19.899235 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:19.899434 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:19.899435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:19.899435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:19.899711 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:19.899879 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:19.899978 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 13:12:19.900099 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260925 13:12:19.907440 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:19.907833 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:19.911957 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: Generated new TSK 0
I20260925 13:12:19.912106 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:19.940312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:20.022558 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:20.022838 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:20.022876 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:20.024605 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:20.024665 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:20.024703 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:20.024739 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:20.026638 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:20.026926 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:20.027165 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:20.032155 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:20.032125 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:20.032167 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:20.032454 1579 server_base.cc:1034] running on GCE node
I20260925 13:12:20.032716 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:20.032992 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:20.034192 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790341940034178 us; error 47 us; skew 500 ppm
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:20.036628 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:20.037055 1579 webserver.cc:460] Webserver started at http://127.0.98.1:34709/ using document root <none> and password file <none>
I20260925 13:12:20.037300 1579 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:20.037367 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:20.037508 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:20.038607 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f3b20af407a4516930e0146b98b333c"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "a4916699364b4ee8d0688028dd93ec10"
server_key_iv: "768b91d6b1a484ec3a432b65fbfd8be3"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.038985 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f3b20af407a4516930e0146b98b333c"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "a4916699364b4ee8d0688028dd93ec10"
server_key_iv: "768b91d6b1a484ec3a432b65fbfd8be3"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.040434 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:12:20.041131 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.041312 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:20.041412 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "4f3b20af407a4516930e0146b98b333c"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "a4916699364b4ee8d0688028dd93ec10"
server_key_iv: "768b91d6b1a484ec3a432b65fbfd8be3"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.041492 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:20.065901 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:20.068265 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:20.068460 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:20.069058 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:20.069139 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.069192 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:20.069224 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:20.077389 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46843
I20260925 13:12:20.077484 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46843 every 8 connection(s)
I20260925 13:12:20.077894 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:20.079965 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.069561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36049 (local address 127.0.98.62:36463)
0925 13:12:20.069636 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.069639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.069838 (+ 199us) server_negotiation.cc:407] Connection header received
0925 13:12:20.070354 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.070355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.070371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.070469 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.071083 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.071899 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.073445 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.073837 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.073931 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.076889 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.076890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.076890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.076898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.078723 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.079140 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.079141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.079141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.079199 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.079398 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.079399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.079399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.079498 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.079546 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.079811 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.079884 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 13:12:20.080173 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.069536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.1:36049)
0925 13:12:20.069700 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.069740 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.070239 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.070531 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.070534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.070774 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.070964 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.070969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.072023 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.072024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.073362 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.073366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.073950 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.073951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.074091 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.074587 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.074598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.076761 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.078817 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.078822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.078824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.079066 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.079246 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.079247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.079247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.079344 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.079562 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.079567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.079690 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.079787 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.079930 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260925 13:12:20.080734 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36463
I20260925 13:12:20.080816 1710 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:20.081032 1710 heartbeater.cc:511] Master 127.0.98.62:36463 requested a full tablet report, sending...
I20260925 13:12:20.081624 1507 ts_manager.cc:194] Registered new tserver with Master: 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843)
I20260925 13:12:20.082669 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36049
I20260925 13:12:20.085608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1579
I20260925 13:12:20.085724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:20.085927 392 external_mini_cluster.cc:1428] Setting key 8ebb4cb31c6164c2fa42aa02f7b9c63a
I20260925 13:12:20.087638 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.081809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.1:44219)
0925 13:12:20.081943 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.081955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.082024 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.082150 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.082151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.082380 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.082454 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.082456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.083090 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.083091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.083703 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.083705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.084032 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.084033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.084083 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.084609 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.084614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.084939 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.086684 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.086685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.086686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.086874 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.087088 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.087089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.087089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.087145 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.087401 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.087402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.087437 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.087498 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.087574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260925 13:12:20.087736 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.081862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44219 (local address 127.0.98.62:36463)
0925 13:12:20.081905 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.081907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.082014 (+ 107us) server_negotiation.cc:407] Connection header received
0925 13:12:20.082090 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.082090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.082097 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.082140 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.082526 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.082999 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.083781 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.083961 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.084016 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.085032 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.085033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.085034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.085038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.086611 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.086962 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.086963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.086964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.087022 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.087205 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.087206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.087206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.087340 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.087404 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.087528 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.087605 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:20.112185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:20.191238 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:20.191424 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:20.191448 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:20.192605 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:20.192643 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:20.192662 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:20.192715 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:20.194271 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:20.194490 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:20.194705 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:20.199975 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:20.199992 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:20.200027 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:20.200181 1719 server_base.cc:1034] running on GCE node
I20260925 13:12:20.200361 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:20.200541 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:20.201738 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790341940201723 us; error 36 us; skew 500 ppm
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:20.204061 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:20.204531 1719 webserver.cc:460] Webserver started at http://127.0.98.2:36049/ using document root <none> and password file <none>
I20260925 13:12:20.204733 1719 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:20.204792 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:20.204948 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:20.206019 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ce46a82f09743b1ae2ad8abad261c42"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "d1093657d715621c0285fd671924980b"
server_key_iv: "4b6056929728d1e5cd8a53510bad8f06"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.206367 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ce46a82f09743b1ae2ad8abad261c42"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "d1093657d715621c0285fd671924980b"
server_key_iv: "4b6056929728d1e5cd8a53510bad8f06"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.207626 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:20.208356 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.208554 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:20.208645 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "5ce46a82f09743b1ae2ad8abad261c42"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "d1093657d715621c0285fd671924980b"
server_key_iv: "4b6056929728d1e5cd8a53510bad8f06"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.208717 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:20.230495 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:20.232414 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:20.232585 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:20.233178 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:20.233230 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.233263 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:20.233285 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:20.240495 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33251
I20260925 13:12:20.240908 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:12:20.240882 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33251 every 8 connection(s)
I20260925 13:12:20.243232 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.233744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60371 (local address 127.0.98.62:36463)
0925 13:12:20.233814 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.233816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.233967 (+ 151us) server_negotiation.cc:407] Connection header received
0925 13:12:20.234577 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.234578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.234593 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.234667 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.235383 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.235945 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.237179 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.237460 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.237558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.240264 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.240265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.240266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.240274 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.241885 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.242320 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.242322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.242322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.242383 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.242635 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.242636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.242636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.242744 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.242815 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.243070 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.243142 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:12:20.243415 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.233568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.2:60371)
0925 13:12:20.233804 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.233859 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.234456 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.234748 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.234753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.235043 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.235244 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.235249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.236069 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.236070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.237054 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.237057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.237585 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.237587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.237771 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.238337 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.238347 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.240135 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.242002 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.242008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.242010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.242226 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.242447 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.242448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.242448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.242568 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.242806 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.242812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.242948 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.243051 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.243199 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
I20260925 13:12:20.243980 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36463
I20260925 13:12:20.244073 1850 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:20.244277 1850 heartbeater.cc:511] Master 127.0.98.62:36463 requested a full tablet report, sending...
I20260925 13:12:20.244740 1507 ts_manager.cc:194] Registered new tserver with Master: 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251)
I20260925 13:12:20.245256 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60371
I20260925 13:12:20.246594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1719
I20260925 13:12:20.246686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:20.246845 392 external_mini_cluster.cc:1428] Setting key fb231c7dfd3f483628afd74d330eb221
I20260925 13:12:20.250937 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.244920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56185 (local address 127.0.98.62:36463)
0925 13:12:20.244972 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.244974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.245075 (+ 101us) server_negotiation.cc:407] Connection header received
0925 13:12:20.245135 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.245136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.245152 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.245195 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.245569 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.246092 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.246918 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.247193 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.247278 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.248183 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.248184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.248185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.248191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.249790 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.250135 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.250136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.250137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.250187 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.250403 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.250404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.250405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.250541 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.250625 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.250777 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.250858 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 13:12:20.250946 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.244885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.2:56185)
0925 13:12:20.245003 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.245014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.245070 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.245201 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.245201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.245404 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.245490 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.245492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.246153 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.246154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.246826 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.246837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.247281 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.247282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.247349 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.247781 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.247786 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.248063 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.249880 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.249881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.249881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.250050 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.250256 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.250257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.250257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.250313 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.250629 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.250630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.250669 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.250752 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.250848 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:20.272217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:20.350629 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:20.350804 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:20.350826 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:20.351893 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:20.351927 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:20.351946 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:20.351963 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:20.353298 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:20.353462 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:20.353636 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:20.358352 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:20.358376 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:20.358417 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:20.358765 1859 server_base.cc:1034] running on GCE node
I20260925 13:12:20.358953 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:20.359211 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:20.360379 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790341940360373 us; error 36 us; skew 500 ppm
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:20.362879 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:20.363315 1859 webserver.cc:460] Webserver started at http://127.0.98.3:41105/ using document root <none> and password file <none>
I20260925 13:12:20.363483 1859 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:20.363520 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:20.363600 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:20.364506 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "85b392a030474f0c87fb02252ba426e1"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "0dcd307b3583bbe2bf403361dc768de4"
server_key_iv: "a288a3499c4e1893f8bb20fddfcd957c"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.364775 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "85b392a030474f0c87fb02252ba426e1"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "0dcd307b3583bbe2bf403361dc768de4"
server_key_iv: "a288a3499c4e1893f8bb20fddfcd957c"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.366009 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:12:20.366755 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.366937 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:20.367003 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "85b392a030474f0c87fb02252ba426e1"
format_stamp: "Formatted at 2026-09-25 13:12:20 on dist-test-slave-2d0s"
server_key: "0dcd307b3583bbe2bf403361dc768de4"
server_key_iv: "a288a3499c4e1893f8bb20fddfcd957c"
server_key_version: "encryptionkey@0"
I20260925 13:12:20.367045 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:20.379585 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:20.381330 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:20.381443 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:20.382038 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:20.382076 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.382098 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:20.382110 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:20.389358 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42343
I20260925 13:12:20.389426 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42343 every 8 connection(s)
I20260925 13:12:20.389784 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:12:20.391996 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.382272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35703 (local address 127.0.98.62:36463)
0925 13:12:20.382321 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.382323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.382549 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:12:20.383288 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.383290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.383302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.383370 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.384223 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.384725 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.385946 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.386199 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.386296 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.389190 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.389192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.389192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.389200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.390824 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.391182 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.391182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.391183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.391237 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.391502 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.391503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.391503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.391610 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.391669 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.391870 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.391932 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:20.392181 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.382250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.3:35703)
0925 13:12:20.382416 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.382457 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.383152 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.383439 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.383443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.383814 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.384110 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.384117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.384883 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.384884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.385839 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.385843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.386342 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.386343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.386487 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.387025 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.387034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.389071 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.390898 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.390902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.390904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.391114 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.391291 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.391291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.391292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.391438 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.391661 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.391664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.391762 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.391836 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.391954 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260925 13:12:20.392650 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36463
I20260925 13:12:20.392729 1990 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:20.392905 1990 heartbeater.cc:511] Master 127.0.98.62:36463 requested a full tablet report, sending...
I20260925 13:12:20.393406 1507 ts_manager.cc:194] Registered new tserver with Master: 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343)
I20260925 13:12:20.393904 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35703
I20260925 13:12:20.396658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 1859
I20260925 13:12:20.396770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:20.396961 392 external_mini_cluster.cc:1428] Setting key 27e71a511fa991c8956a194bf65ca7ce
I20260925 13:12:20.397891 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:12:20.399549 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.393718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43523 (local address 127.0.98.62:36463)
0925 13:12:20.394201 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.394203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.394211 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:20.394226 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.394227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.394235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.394279 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.394617 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.395063 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.395735 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.395917 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.396019 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.397065 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.397066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.397067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.397074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.398542 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.398929 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.398929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.398930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.398973 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.399152 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.399152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.399153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.399232 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.399275 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.399424 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.399475 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467}
I20260925 13:12:20.399686 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.393636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.3:43523)
0925 13:12:20.393759 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.393769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.393821 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.394302 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.394303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.394493 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.394544 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.394546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.395147 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.395147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.395646 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.395647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.395989 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.395990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.396035 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.396547 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.396552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.396942 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.398623 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.398624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.398625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.398858 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.399034 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.399035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.399035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.399099 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.399286 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.399287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.399328 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.399407 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.399532 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:20.430616 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.422715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.0.98.62:36463)
0925 13:12:20.422786 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.422788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.422796 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:20.422821 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.422822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.422831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.422905 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.423491 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.424307 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.425113 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.425480 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.425562 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.428050 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.428051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.428051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.428057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.429391 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.429754 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.429755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.429755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.429801 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.429982 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.429983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.429983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.430086 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.430157 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.430396 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.430510 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:12:20.432437 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:20.434106 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:20.440555 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.436469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.0.1:48554)
0925 13:12:20.436570 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.436613 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.436683 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.437356 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.437357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.437360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.437418 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.437420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.438469 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.438470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.439478 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.439481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.440259 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.440260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.440338 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.440352 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.440432 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.440479 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":60,"threads_started":1}
I20260925 13:12:20.440590 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.436769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.98.3:42343)
0925 13:12:20.437179 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.437183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.437193 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:20.437230 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.437231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.437249 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.437500 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.437562 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.438354 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.439579 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.440170 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.440251 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.440334 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.440449 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.440486 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":99,"threads_started":1}
I20260925 13:12:20.441166 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.436508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46843 (local address 127.0.0.1:44288)
0925 13:12:20.436641 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.436650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.436689 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.437394 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.437395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.437396 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.437425 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.437427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.438722 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.438723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.439965 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.439966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.440915 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.440916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.440977 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.440982 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.441046 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.441086 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":45,"thread_start_us":98,"threads_started":1}
I20260925 13:12:20.441191 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.436964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.0.98.1:46843)
0925 13:12:20.437135 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.437140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.437152 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:12:20.437198 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.437201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.437217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.437325 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.437510 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.438641 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.440057 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.440822 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.440913 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.440985 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.441058 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.441093 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260925 13:12:20.441368 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 1357a7d4036843d5be92838333c962bb (DEFAULT_TABLE table=test-table [id=cb9772e3cf4345dc9127a59b46299483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:20.441777 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1357a7d4036843d5be92838333c962bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.441823 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 1357a7d4036843d5be92838333c962bb (DEFAULT_TABLE table=test-table [id=cb9772e3cf4345dc9127a59b46299483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:20.442083 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1357a7d4036843d5be92838333c962bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.444779 2013 tablet_bootstrap.cc:492] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Bootstrap starting.
I20260925 13:12:20.445371 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.436508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33251 (local address 127.0.0.1:46188)
0925 13:12:20.436782 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.436790 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.436821 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.437219 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.437224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.437226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.437295 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.437297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.438706 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.438707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.439964 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.439966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.445111 (+ 5145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.445112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.445171 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.445175 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.445250 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.445292 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":72,"thread_start_us":88,"threads_started":1}
I20260925 13:12:20.445555 2013 tablet_bootstrap.cc:654] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.445655 2014 tablet_bootstrap.cc:492] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Bootstrap starting.
I20260925 13:12:20.445658 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.436773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.0.98.2:33251)
0925 13:12:20.436959 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.436963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.436972 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:20.437002 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.437003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.437016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.437102 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.437401 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.438610 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.440063 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.444947 (+ 4884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.445009 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.445081 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.445268 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.445309 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260925 13:12:20.446178 2013 log.cc:824] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:20.446342 2014 tablet_bootstrap.cc:654] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.446352 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 1357a7d4036843d5be92838333c962bb (DEFAULT_TABLE table=test-table [id=cb9772e3cf4345dc9127a59b46299483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:20.446640 2014 log.cc:824] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:20.446731 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1357a7d4036843d5be92838333c962bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.447338 2014 tablet_bootstrap.cc:492] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: No bootstrap required, opened a new log
I20260925 13:12:20.447394 2014 ts_tablet_manager.cc:1397] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:12:20.448843 2014 raft_consensus.cc:348] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.448975 2014 raft_consensus.cc:374] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.449018 2014 raft_consensus.cc:729] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3b20af407a4516930e0146b98b333c, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.449147 2014 consensus_queue.cc:260] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.449452 2014 ts_tablet_manager.cc:1428] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:12:20.449522 1710 heartbeater.cc:503] Master 127.0.98.62:36463 was elected leader, sending a full tablet report...
I20260925 13:12:20.450001 2013 tablet_bootstrap.cc:492] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: No bootstrap required, opened a new log
I20260925 13:12:20.450078 2013 ts_tablet_manager.cc:1397] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 13:12:20.450021 2018 tablet_bootstrap.cc:492] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Bootstrap starting.
I20260925 13:12:20.450628 2018 tablet_bootstrap.cc:654] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.450922 2018 log.cc:824] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:20.451582 2018 tablet_bootstrap.cc:492] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: No bootstrap required, opened a new log
I20260925 13:12:20.451673 2018 ts_tablet_manager.cc:1397] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:20.451969 2013 raft_consensus.cc:348] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.452152 2013 raft_consensus.cc:374] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.452206 2013 raft_consensus.cc:729] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b392a030474f0c87fb02252ba426e1, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.452340 2013 consensus_queue.cc:260] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.452608 2013 ts_tablet_manager.cc:1428] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 13:12:20.452644 1990 heartbeater.cc:503] Master 127.0.98.62:36463 was elected leader, sending a full tablet report...
I20260925 13:12:20.453845 2018 raft_consensus.cc:348] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.453953 2018 raft_consensus.cc:374] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.453974 2018 raft_consensus.cc:729] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce46a82f09743b1ae2ad8abad261c42, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.454066 2018 consensus_queue.cc:260] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.454272 2018 ts_tablet_manager.cc:1428] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:12:20.454360 1850 heartbeater.cc:503] Master 127.0.98.62:36463 was elected leader, sending a full tablet report...
W20260925 13:12:20.491578 1851 tablet.cc:2369] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:12:20.578786 1711 tablet.cc:2369] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:20.613072 2020 raft_consensus.cc:482] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:20.613238 2020 raft_consensus.cc:504] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.613685 2020 leader_election.cc:288] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251), 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843)
I20260925 13:12:20.616771 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.613967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34067 (local address 127.0.98.2:33251)
0925 13:12:20.614016 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.614018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.614030 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:12:20.614130 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.614131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.614143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.614201 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.614510 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.615073 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.616030 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.616405 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.616498 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.616509 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.616666 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.616705 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:12:20.616786 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.613890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33251 (local address 127.0.98.3:34067)
0925 13:12:20.613954 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.613968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.614023 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.614222 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.614223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.614226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.614304 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.614306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.615161 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.615162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.615933 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.615935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.616497 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.616498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.616575 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.616583 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.616657 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.616698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 13:12:20.616885 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.614017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44497 (local address 127.0.98.1:46843)
0925 13:12:20.614087 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.614089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.614256 (+ 167us) server_negotiation.cc:407] Connection header received
0925 13:12:20.614391 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.614392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.614417 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.614469 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.614639 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.615126 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.616101 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.616475 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.616565 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.616575 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.616767 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.616815 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 13:12:20.616957 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.613916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46843 (local address 127.0.98.3:44497)
0925 13:12:20.614200 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.614210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.614266 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.614493 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.614493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.614495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.614555 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.614556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.615226 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.615227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.615998 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.616001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.616568 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.616569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.616643 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.616652 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.616743 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.616786 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":60,"thread_start_us":116,"threads_started":1}
I20260925 13:12:20.617350 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1357a7d4036843d5be92838333c962bb" candidate_uuid: "85b392a030474f0c87fb02252ba426e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3b20af407a4516930e0146b98b333c" is_pre_election: true
I20260925 13:12:20.617369 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1357a7d4036843d5be92838333c962bb" candidate_uuid: "85b392a030474f0c87fb02252ba426e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce46a82f09743b1ae2ad8abad261c42" is_pre_election: true
I20260925 13:12:20.617532 1664 raft_consensus.cc:2454] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b392a030474f0c87fb02252ba426e1 in term 0.
I20260925 13:12:20.617532 1804 raft_consensus.cc:2454] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b392a030474f0c87fb02252ba426e1 in term 0.
I20260925 13:12:20.617873 1877 leader_election.cc:302] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f3b20af407a4516930e0146b98b333c, 5ce46a82f09743b1ae2ad8abad261c42, 85b392a030474f0c87fb02252ba426e1; no voters:
I20260925 13:12:20.618041 2020 raft_consensus.cc:2781] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:20.618131 2020 raft_consensus.cc:482] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:20.618161 2020 raft_consensus.cc:3037] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:20.619182 2020 raft_consensus.cc:504] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.619367 2020 leader_election.cc:288] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [CANDIDATE]: Term 1 election: Requested vote from peers 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251), 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843)
I20260925 13:12:20.619493 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1357a7d4036843d5be92838333c962bb" candidate_uuid: "85b392a030474f0c87fb02252ba426e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce46a82f09743b1ae2ad8abad261c42"
I20260925 13:12:20.619532 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1357a7d4036843d5be92838333c962bb" candidate_uuid: "85b392a030474f0c87fb02252ba426e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3b20af407a4516930e0146b98b333c"
I20260925 13:12:20.619598 1804 raft_consensus.cc:3037] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:20.619613 1664 raft_consensus.cc:3037] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:20.620292 1804 raft_consensus.cc:2454] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b392a030474f0c87fb02252ba426e1 in term 1.
I20260925 13:12:20.620357 1664 raft_consensus.cc:2454] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b392a030474f0c87fb02252ba426e1 in term 1.
I20260925 13:12:20.620573 1877 leader_election.cc:302] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3b20af407a4516930e0146b98b333c, 85b392a030474f0c87fb02252ba426e1; no voters:
I20260925 13:12:20.620684 2020 raft_consensus.cc:2781] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:20.620890 2020 raft_consensus.cc:686] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 1 LEADER]: Becoming Leader. State: Replica: 85b392a030474f0c87fb02252ba426e1, State: Running, Role: LEADER
I20260925 13:12:20.621008 2020 consensus_queue.cc:237] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } }
I20260925 13:12:20.622009 1507 catalog_manager.cc:5495] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b392a030474f0c87fb02252ba426e1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "85b392a030474f0c87fb02252ba426e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } health_report { overall_health: UNKNOWN } } }
W20260925 13:12:20.640345 1991 tablet.cc:2369] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:20.667989 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.665437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.0.98.3:42343)
0925 13:12:20.665515 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.665517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.665528 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:20.665635 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.665636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.665682 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.665758 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:20.665919 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.666384 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.667179 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.667375 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.667451 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.667646 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:20.667747 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:20.667846 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.667875 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.667897 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:12:20.669975 1664 raft_consensus.cc:1261] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Refusing update from remote peer 85b392a030474f0c87fb02252ba426e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:20.669975 1804 raft_consensus.cc:1261] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Refusing update from remote peer 85b392a030474f0c87fb02252ba426e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:20.670368 2020 consensus_queue.cc:1035] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:20.670545 2023 consensus_queue.cc:1035] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:20.673532 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333240588980932608 to 7333240588784570368. Current Snapshot: MvccSnapshot[applied={T|T < 7333240588980932608}]
I20260925 13:12:20.673534 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333240588980932608 to 7333240588784570368. Current Snapshot: MvccSnapshot[applied={T|T < 7333240588980932608}]
I20260925 13:12:20.676956 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:20.677059 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:20.677810 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: Sending DeleteTablet for 3 replicas of tablet 1357a7d4036843d5be92838333c962bb
I20260925 13:12:20.678244 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1357a7d4036843d5be92838333c962bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44288
I20260925 13:12:20.678272 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 1357a7d4036843d5be92838333c962bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46188
I20260925 13:12:20.678481 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 1357a7d4036843d5be92838333c962bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48554
I20260925 13:12:20.678483 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:20.678628 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:20.678701 2037 tablet_replica.cc:330] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: stopping tablet replica
I20260925 13:12:20.678933 2037 raft_consensus.cc:2229] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:20.679059 2037 raft_consensus.cc:2258] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:20.679289 2036 tablet_replica.cc:330] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: stopping tablet replica
I20260925 13:12:20.679396 2036 raft_consensus.cc:2229] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:20.679492 2036 raft_consensus.cc:2258] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:20.679725 2037 ts_tablet_manager.cc:1905] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:20.679877 2036 ts_tablet_manager.cc:1905] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:20.680075 1784 tablet_service.cc:1467] Processing CreateTablet for tablet dfac8253f8d549b4b0c26704148821a8 (DEFAULT_TABLE table=test-table [id=22a7cc876f084d6383631fac98c48742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:20.680186 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac8253f8d549b4b0c26704148821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.681530 2036 ts_tablet_manager.cc:1918] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:20.681530 2037 ts_tablet_manager.cc:1918] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:20.681531 2018 tablet_bootstrap.cc:492] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Bootstrap starting.
I20260925 13:12:20.681596 2036 log.cc:1196] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/1357a7d4036843d5be92838333c962bb
I20260925 13:12:20.681631 2037 log.cc:1196] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/1357a7d4036843d5be92838333c962bb
I20260925 13:12:20.681910 2036 ts_tablet_manager.cc:1939] T 1357a7d4036843d5be92838333c962bb P 4f3b20af407a4516930e0146b98b333c: Deleting consensus metadata
I20260925 13:12:20.681910 2037 ts_tablet_manager.cc:1939] T 1357a7d4036843d5be92838333c962bb P 5ce46a82f09743b1ae2ad8abad261c42: Deleting consensus metadata
I20260925 13:12:20.682027 2018 tablet_bootstrap.cc:654] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.682404 1494 catalog_manager.cc:4841] TS 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251): tablet 1357a7d4036843d5be92838333c962bb (table test-table [id=cb9772e3cf4345dc9127a59b46299483]) successfully deleted
I20260925 13:12:20.682651 1644 tablet_service.cc:1467] Processing CreateTablet for tablet dfac8253f8d549b4b0c26704148821a8 (DEFAULT_TABLE table=test-table [id=22a7cc876f084d6383631fac98c48742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:20.682672 1924 tablet_service.cc:1467] Processing CreateTablet for tablet dfac8253f8d549b4b0c26704148821a8 (DEFAULT_TABLE table=test-table [id=22a7cc876f084d6383631fac98c48742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:20.682742 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac8253f8d549b4b0c26704148821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.682767 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac8253f8d549b4b0c26704148821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.683627 2013 tablet_bootstrap.cc:492] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Bootstrap starting.
I20260925 13:12:20.683866 2014 tablet_bootstrap.cc:492] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Bootstrap starting.
I20260925 13:12:20.683959 2013 tablet_bootstrap.cc:654] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.684221 2014 tablet_bootstrap.cc:654] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.684513 2013 tablet_bootstrap.cc:492] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: No bootstrap required, opened a new log
I20260925 13:12:20.684501 1493 catalog_manager.cc:4841] TS 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843): tablet 1357a7d4036843d5be92838333c962bb (table test-table [id=cb9772e3cf4345dc9127a59b46299483]) successfully deleted
I20260925 13:12:20.684556 2013 ts_tablet_manager.cc:1397] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:20.684726 2013 raft_consensus.cc:348] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.684782 2013 raft_consensus.cc:374] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.684796 2013 raft_consensus.cc:729] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b392a030474f0c87fb02252ba426e1, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.684824 2013 consensus_queue.cc:260] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.684918 2013 ts_tablet_manager.cc:1428] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.685297 2038 tablet_replica.cc:330] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: stopping tablet replica
I20260925 13:12:20.685333 2014 tablet_bootstrap.cc:492] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: No bootstrap required, opened a new log
I20260925 13:12:20.685367 2014 ts_tablet_manager.cc:1397] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:12:20.685408 2038 raft_consensus.cc:2229] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:20.685504 2038 raft_consensus.cc:2258] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:20.685490 2014 raft_consensus.cc:348] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.685560 2014 raft_consensus.cc:374] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.685588 2014 raft_consensus.cc:729] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3b20af407a4516930e0146b98b333c, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.685624 2014 consensus_queue.cc:260] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.685731 2014 ts_tablet_manager.cc:1428] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.686303 2038 ts_tablet_manager.cc:1905] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:20.687729 2038 ts_tablet_manager.cc:1918] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:20.687798 2038 log.cc:1196] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/1357a7d4036843d5be92838333c962bb
I20260925 13:12:20.688040 2038 ts_tablet_manager.cc:1939] T 1357a7d4036843d5be92838333c962bb P 85b392a030474f0c87fb02252ba426e1: Deleting consensus metadata
I20260925 13:12:20.688431 1491 catalog_manager.cc:4841] TS 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343): tablet 1357a7d4036843d5be92838333c962bb (table test-table [id=cb9772e3cf4345dc9127a59b46299483]) successfully deleted
I20260925 13:12:20.689052 2018 tablet_bootstrap.cc:492] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: No bootstrap required, opened a new log
I20260925 13:12:20.689093 2018 ts_tablet_manager.cc:1397] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 13:12:20.689214 2018 raft_consensus.cc:348] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.689265 2018 raft_consensus.cc:374] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.689276 2018 raft_consensus.cc:729] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce46a82f09743b1ae2ad8abad261c42, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.689359 2018 consensus_queue.cc:260] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.689483 2018 ts_tablet_manager.cc:1428] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:20.879853 2021 raft_consensus.cc:482] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:20.879969 2021 raft_consensus.cc:504] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.880417 2021 leader_election.cc:288] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843), 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343)
I20260925 13:12:20.883414 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.880819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53037 (local address 127.0.98.1:46843)
0925 13:12:20.880884 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.880886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.880894 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:20.880929 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.880930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.880937 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.881013 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.881244 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.881773 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.882742 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.883083 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.883134 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.883165 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.883307 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.883347 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 13:12:20.883451 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.880682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43921 (local address 127.0.98.3:42343)
0925 13:12:20.880733 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.880735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.880850 (+ 115us) server_negotiation.cc:407] Connection header received
0925 13:12:20.880898 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.880899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.880916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.880960 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:20.881202 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.881725 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.882740 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.883091 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.883176 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.883187 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.883351 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.883391 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:20.883450 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.880609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46843 (local address 127.0.98.2:53037)
0925 13:12:20.880749 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.880760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.880843 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.881029 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.881030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.881033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.881157 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.881159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.881860 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.881860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.882631 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.882633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.883159 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.883160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.883231 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.883235 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.883283 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.883341 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 13:12:20.883458 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.880609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.98.2:43921)
0925 13:12:20.880757 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.880768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.880835 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.880987 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.880988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.880991 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:20.881073 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.881075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.881796 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.881797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.882633 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.882635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.883192 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.883193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.883253 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.883256 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.883335 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.883373 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":59,"thread_start_us":73,"threads_started":1}
I20260925 13:12:20.883670 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac8253f8d549b4b0c26704148821a8" candidate_uuid: "5ce46a82f09743b1ae2ad8abad261c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3b20af407a4516930e0146b98b333c" is_pre_election: true
I20260925 13:12:20.883766 1664 raft_consensus.cc:2454] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce46a82f09743b1ae2ad8abad261c42 in term 0.
I20260925 13:12:20.883821 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac8253f8d549b4b0c26704148821a8" candidate_uuid: "5ce46a82f09743b1ae2ad8abad261c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b392a030474f0c87fb02252ba426e1" is_pre_election: true
I20260925 13:12:20.883947 1944 raft_consensus.cc:2454] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce46a82f09743b1ae2ad8abad261c42 in term 0.
I20260925 13:12:20.883949 1737 leader_election.cc:302] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3b20af407a4516930e0146b98b333c, 5ce46a82f09743b1ae2ad8abad261c42; no voters:
I20260925 13:12:20.884054 2021 raft_consensus.cc:2781] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:20.884093 2021 raft_consensus.cc:482] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:20.884117 2021 raft_consensus.cc:3037] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:20.884779 2021 raft_consensus.cc:504] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.884960 2021 leader_election.cc:288] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [CANDIDATE]: Term 1 election: Requested vote from peers 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843), 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343)
I20260925 13:12:20.885150 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac8253f8d549b4b0c26704148821a8" candidate_uuid: "5ce46a82f09743b1ae2ad8abad261c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b392a030474f0c87fb02252ba426e1"
I20260925 13:12:20.885161 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac8253f8d549b4b0c26704148821a8" candidate_uuid: "5ce46a82f09743b1ae2ad8abad261c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3b20af407a4516930e0146b98b333c"
I20260925 13:12:20.885227 1664 raft_consensus.cc:3037] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:20.885227 1944 raft_consensus.cc:3037] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:20.886133 1944 raft_consensus.cc:2454] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce46a82f09743b1ae2ad8abad261c42 in term 1.
I20260925 13:12:20.886387 1664 raft_consensus.cc:2454] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce46a82f09743b1ae2ad8abad261c42 in term 1.
I20260925 13:12:20.886615 1735 leader_election.cc:302] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce46a82f09743b1ae2ad8abad261c42, 85b392a030474f0c87fb02252ba426e1; no voters:
I20260925 13:12:20.886737 2021 raft_consensus.cc:2781] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:20.886870 2021 raft_consensus.cc:686] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce46a82f09743b1ae2ad8abad261c42, State: Running, Role: LEADER
I20260925 13:12:20.886950 2021 consensus_queue.cc:237] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.887651 1507 catalog_manager.cc:5495] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ce46a82f09743b1ae2ad8abad261c42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: UNKNOWN } } }
Sep 25 13:12:20 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:20.949875 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.942506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.0.98.62:36463)
0925 13:12:20.942613 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:20.942615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:20.942642 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:12:20.942717 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:20.942718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:20.942728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:20.942786 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:20.943269 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.943869 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.944660 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.944844 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.944937 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.947075 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:20.947076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:20.947076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:20.947083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:20.948725 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.949084 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.949085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.949086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.949147 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:20.949364 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:20.949365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:20.949365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:20.949474 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:20.949531 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:20.949724 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 13:12:20.949801 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
I20260925 13:12:20.950026 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:20.942429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.0.1:45412)
0925 13:12:20.942565 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:20.942579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:20.942659 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:20.942804 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:20.942804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:20.943098 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:20.943143 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.943145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.943946 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.943947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.944574 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:20.944576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:20.944935 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:20.944936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:20.944987 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:20.945545 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:20.945549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:20.946979 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:20.948804 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.948810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.948812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.949017 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.949226 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:20.949227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:20.949228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:20.949295 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:20.949533 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:20.949539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:20.949582 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:12:20.949651 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:12:20.949879 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":66,"thread_start_us":56,"threads_started":1}
I20260925 13:12:20.952008 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 13:12:20.952232 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:20.953951 1644 tablet_service.cc:1467] Processing CreateTablet for tablet d2ab16e64efa4a568f07cf491f2f26eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd734e5fc5184f79aef994b33166843d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:20.953945 1784 tablet_service.cc:1467] Processing CreateTablet for tablet d2ab16e64efa4a568f07cf491f2f26eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd734e5fc5184f79aef994b33166843d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:20.954118 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ab16e64efa4a568f07cf491f2f26eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.954119 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ab16e64efa4a568f07cf491f2f26eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.955412 2014 tablet_bootstrap.cc:492] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: Bootstrap starting.
I20260925 13:12:20.955505 2018 tablet_bootstrap.cc:492] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: Bootstrap starting.
I20260925 13:12:20.955551 1924 tablet_service.cc:1467] Processing CreateTablet for tablet d2ab16e64efa4a568f07cf491f2f26eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd734e5fc5184f79aef994b33166843d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:20.955690 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ab16e64efa4a568f07cf491f2f26eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:20.955806 2014 tablet_bootstrap.cc:654] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.955979 2018 tablet_bootstrap.cc:654] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.956401 2014 tablet_bootstrap.cc:492] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: No bootstrap required, opened a new log
I20260925 13:12:20.956454 2014 ts_tablet_manager.cc:1397] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:20.956602 2014 raft_consensus.cc:348] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.956656 2014 raft_consensus.cc:374] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.956668 2014 raft_consensus.cc:729] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3b20af407a4516930e0146b98b333c, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.956670 2018 tablet_bootstrap.cc:492] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: No bootstrap required, opened a new log
I20260925 13:12:20.956724 2018 ts_tablet_manager.cc:1397] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:20.956728 2014 consensus_queue.cc:260] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.956818 2014 ts_tablet_manager.cc:1428] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.956883 2018 raft_consensus.cc:348] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.956900 2017 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:20.956964 2018 raft_consensus.cc:374] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.956967 2017 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:20.956995 2018 raft_consensus.cc:729] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce46a82f09743b1ae2ad8abad261c42, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.957046 2018 consensus_queue.cc:260] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.957159 2018 ts_tablet_manager.cc:1428] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.957300 2021 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:20.957376 2021 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:20.957463 2013 tablet_bootstrap.cc:492] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: Bootstrap starting.
I20260925 13:12:20.957923 2013 tablet_bootstrap.cc:654] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:20.958492 2013 tablet_bootstrap.cc:492] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: No bootstrap required, opened a new log
I20260925 13:12:20.958539 2013 ts_tablet_manager.cc:1397] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:20.958825 2013 raft_consensus.cc:348] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.958891 2013 raft_consensus.cc:374] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:20.958905 2013 raft_consensus.cc:729] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b392a030474f0c87fb02252ba426e1, State: Initialized, Role: FOLLOWER
I20260925 13:12:20.958976 2013 consensus_queue.cc:260] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:20.959076 2013 ts_tablet_manager.cc:1428] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:20.959090 2023 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:20.959159 2023 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:21.037141 2017 raft_consensus.cc:482] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:21.037250 2017 raft_consensus.cc:504] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:21.037603 2017 leader_election.cc:288] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251), 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343)
I20260925 13:12:21.041201 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.037970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42011 (local address 127.0.98.2:33251)
0925 13:12:21.038017 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.038019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.038155 (+ 136us) server_negotiation.cc:407] Connection header received
0925 13:12:21.038239 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.038240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.038252 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.038300 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:21.038474 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.038952 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.040474 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.040827 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.040915 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.040926 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.041083 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.041123 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:12:21.041245 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.037923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48355 (local address 127.0.98.3:42343)
0925 13:12:21.037995 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.037997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.038163 (+ 166us) server_negotiation.cc:407] Connection header received
0925 13:12:21.038248 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.038249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.038264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.038360 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:21.038529 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.039004 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.040484 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.040858 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.040956 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.040966 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.041128 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.041174 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 13:12:21.041262 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.037878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33251 (local address 127.0.98.1:42011)
0925 13:12:21.038079 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:21.038088 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:21.038171 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:21.038315 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:21.038316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:21.038318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:21.038403 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.038405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.039035 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.039036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.040367 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.040370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.040913 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.040914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.040994 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.040997 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:21.041073 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:12:21.041117 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":46,"thread_start_us":123,"threads_started":1}
I20260925 13:12:21.041501 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.037859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.98.1:48355)
0925 13:12:21.038062 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:21.038076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:21.038176 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:21.038363 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:21.038364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:21.038366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:21.038452 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.038454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.039099 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.039100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.040370 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.040373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.040958 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.040959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.041036 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.041039 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:21.041122 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 13:12:21.041163 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":424,"thread_start_us":129,"threads_started":1}
I20260925 13:12:21.042119 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ab16e64efa4a568f07cf491f2f26eb" candidate_uuid: "4f3b20af407a4516930e0146b98b333c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce46a82f09743b1ae2ad8abad261c42" is_pre_election: true
I20260925 13:12:21.042233 1804 raft_consensus.cc:2454] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f3b20af407a4516930e0146b98b333c in term 0.
I20260925 13:12:21.042270 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ab16e64efa4a568f07cf491f2f26eb" candidate_uuid: "4f3b20af407a4516930e0146b98b333c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b392a030474f0c87fb02252ba426e1" is_pre_election: true
I20260925 13:12:21.042380 1944 raft_consensus.cc:2454] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f3b20af407a4516930e0146b98b333c in term 0.
I20260925 13:12:21.042456 1598 leader_election.cc:302] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3b20af407a4516930e0146b98b333c, 5ce46a82f09743b1ae2ad8abad261c42; no voters:
I20260925 13:12:21.042618 2017 raft_consensus.cc:2781] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:21.042659 2017 raft_consensus.cc:482] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:21.042686 2017 raft_consensus.cc:3037] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:21.043510 2017 raft_consensus.cc:504] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:21.043648 2017 leader_election.cc:288] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [CANDIDATE]: Term 1 election: Requested vote from peers 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251), 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343)
I20260925 13:12:21.043809 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ab16e64efa4a568f07cf491f2f26eb" candidate_uuid: "4f3b20af407a4516930e0146b98b333c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b392a030474f0c87fb02252ba426e1"
I20260925 13:12:21.043820 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ab16e64efa4a568f07cf491f2f26eb" candidate_uuid: "4f3b20af407a4516930e0146b98b333c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce46a82f09743b1ae2ad8abad261c42"
I20260925 13:12:21.043892 1944 raft_consensus.cc:3037] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:21.043912 1804 raft_consensus.cc:3037] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:21.044585 1944 raft_consensus.cc:2454] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f3b20af407a4516930e0146b98b333c in term 1.
I20260925 13:12:21.044760 1804 raft_consensus.cc:2454] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f3b20af407a4516930e0146b98b333c in term 1.
I20260925 13:12:21.044756 1595 leader_election.cc:302] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3b20af407a4516930e0146b98b333c, 85b392a030474f0c87fb02252ba426e1; no voters:
I20260925 13:12:21.044844 2017 raft_consensus.cc:2781] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:21.044999 2017 raft_consensus.cc:686] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [term 1 LEADER]: Becoming Leader. State: Replica: 4f3b20af407a4516930e0146b98b333c, State: Running, Role: LEADER
I20260925 13:12:21.045069 2017 consensus_queue.cc:237] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260925 13:12:21.045449 2058 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: TxnStatusTablet state changed. Reason: New leader 4f3b20af407a4516930e0146b98b333c. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.045544 2058 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:21.045733 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 13:12:21.045889 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 13:12:21.045929 1507 catalog_manager.cc:5495] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f3b20af407a4516930e0146b98b333c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:21.051239 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.048848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.0.98.1:46843)
0925 13:12:21.048894 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.048896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.048905 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:21.048960 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.048960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.048978 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.049040 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:21.049163 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.049634 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.050444 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.050673 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.050744 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.050915 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.051012 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.051103 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.051159 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.051177 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:12:21.051275 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.048760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46843 (local address 127.0.0.1:44294)
0925 13:12:21.048833 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:21.048844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:21.048886 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:21.049056 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:21.049057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:21.049058 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:21.049099 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.049102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.049760 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.049761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.050349 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.050351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.050743 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.050744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.050800 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.050815 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.051091 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.051100 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 13:12:21.051154 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:12:21.051204 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 13:12:21.052177 1804 raft_consensus.cc:1261] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Refusing update from remote peer 4f3b20af407a4516930e0146b98b333c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:21.052269 1944 raft_consensus.cc:1261] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Refusing update from remote peer 4f3b20af407a4516930e0146b98b333c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:21.052470 2058 consensus_queue.cc:1035] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:21.052611 2017 consensus_queue.cc:1035] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:21.053113 2021 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: TxnStatusTablet state changed. Reason: New leader 4f3b20af407a4516930e0146b98b333c. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.053202 2021 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:21.053171 2023 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: TxnStatusTablet state changed. Reason: New leader 4f3b20af407a4516930e0146b98b333c. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.053257 2023 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:21.053843 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333240590548180992 to 7333240590521151488. Current Snapshot: MvccSnapshot[applied={T|T < 7333240590548180992}]
I20260925 13:12:21.053910 2017 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.054015 2017 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:21.054157 2023 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.054229 2023 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 85b392a030474f0c87fb02252ba426e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:21.055580 2058 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.055675 2058 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 4f3b20af407a4516930e0146b98b333c: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:21.055751 2021 tablet_replica.cc:439] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f3b20af407a4516930e0146b98b333c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 } } peers { permanent_uuid: "5ce46a82f09743b1ae2ad8abad261c42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260925 13:12:21.055866 2021 tablet_replica.cc:442] T d2ab16e64efa4a568f07cf491f2f26eb P 5ce46a82f09743b1ae2ad8abad261c42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:21.059371 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.056665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.98.2:33251)
0925 13:12:21.056710 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.056713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.056724 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:21.056819 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.056820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.056854 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.056933 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:21.057063 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.057805 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.058480 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.058768 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.058861 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.058990 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.059118 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.059198 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.059271 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.059291 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:21.065317 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.062102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54039 (local address 127.0.98.1:46843)
0925 13:12:21.062157 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.062159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.062253 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:12:21.062398 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.062399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.062410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.062488 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:21.062989 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.063524 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.064663 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.064846 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.064901 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.065097 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.065130 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.065184 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.065254 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.065268 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:21.065550 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.062030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46843 (local address 127.0.98.2:54039)
0925 13:12:21.062195 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:21.062206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:21.062270 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:21.062821 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:21.062822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:21.062824 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:21.062913 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.062915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.063625 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.063626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.064581 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.064583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.064932 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.064934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.064998 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.065011 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.065200 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.065202 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:12:21.065252 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:12:21.065293 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260925 13:12:21.067909 1944 raft_consensus.cc:1261] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Refusing update from remote peer 5ce46a82f09743b1ae2ad8abad261c42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:21.067975 1664 raft_consensus.cc:1261] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Refusing update from remote peer 5ce46a82f09743b1ae2ad8abad261c42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:21.068174 2044 consensus_queue.cc:1035] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b392a030474f0c87fb02252ba426e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:21.068315 2021 consensus_queue.cc:1035] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3b20af407a4516930e0146b98b333c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:21.077769 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.074787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55833 (local address 127.0.98.2:33251)
0925 13:12:21.074853 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.074855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.075004 (+ 149us) server_negotiation.cc:407] Connection header received
0925 13:12:21.075068 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.075069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.075080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.075132 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:21.075280 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.075875 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.076968 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.077214 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.077273 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.077491 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.077525 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.077551 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.077654 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.077703 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:21.078004 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.074760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33251 (local address 127.0.98.1:55833)
0925 13:12:21.074951 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:21.074962 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:21.075019 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:21.075136 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:21.075137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:21.075138 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:21.075200 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.075201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.075956 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.075957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.076877 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:21.076880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.077307 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.077308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:21.077388 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.077402 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:21.077592 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:21.077593 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:12:21.077861 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 13:12:21.077908 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260925 13:12:21.090965 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:21.091118 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:21.091825 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a705a833884b869aecac965319f6f2: Sending DeleteTablet for 3 replicas of tablet dfac8253f8d549b4b0c26704148821a8
I20260925 13:12:21.092089 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac8253f8d549b4b0c26704148821a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44288
I20260925 13:12:21.092175 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac8253f8d549b4b0c26704148821a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48554
I20260925 13:12:21.092216 2036 tablet_replica.cc:330] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: stopping tablet replica
I20260925 13:12:21.092258 2036 raft_consensus.cc:2229] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:21.092275 2038 tablet_replica.cc:330] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: stopping tablet replica
I20260925 13:12:21.092306 2036 raft_consensus.cc:2258] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:21.092347 2038 raft_consensus.cc:2229] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:21.092382 2038 raft_consensus.cc:2258] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:21.092641 2036 ts_tablet_manager.cc:1905] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:21.092718 2038 ts_tablet_manager.cc:1905] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:21.094521 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac8253f8d549b4b0c26704148821a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46188
I20260925 13:12:21.094614 2037 tablet_replica.cc:330] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: stopping tablet replica
I20260925 13:12:21.094622 2038 ts_tablet_manager.cc:1918] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:21.094657 2037 raft_consensus.cc:2229] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:21.094674 2038 log.cc:1196] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/dfac8253f8d549b4b0c26704148821a8
I20260925 13:12:21.094750 2037 raft_consensus.cc:2258] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:21.094920 2038 ts_tablet_manager.cc:1939] T dfac8253f8d549b4b0c26704148821a8 P 85b392a030474f0c87fb02252ba426e1: Deleting consensus metadata
I20260925 13:12:21.095089 2037 ts_tablet_manager.cc:1905] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:21.095219 1491 catalog_manager.cc:4841] TS 85b392a030474f0c87fb02252ba426e1 (127.0.98.3:42343): tablet dfac8253f8d549b4b0c26704148821a8 (table test-table [id=22a7cc876f084d6383631fac98c48742]) successfully deleted
I20260925 13:12:21.096807 2037 ts_tablet_manager.cc:1918] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:21.096858 2037 log.cc:1196] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/dfac8253f8d549b4b0c26704148821a8
I20260925 13:12:21.097064 2037 ts_tablet_manager.cc:1939] T dfac8253f8d549b4b0c26704148821a8 P 5ce46a82f09743b1ae2ad8abad261c42: Deleting consensus metadata
I20260925 13:12:21.097512 1494 catalog_manager.cc:4841] TS 5ce46a82f09743b1ae2ad8abad261c42 (127.0.98.2:33251): tablet dfac8253f8d549b4b0c26704148821a8 (table test-table [id=22a7cc876f084d6383631fac98c48742]) successfully deleted
I20260925 13:12:21.097811 2036 ts_tablet_manager.cc:1918] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:21.097875 2036 log.cc:1196] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/dfac8253f8d549b4b0c26704148821a8
I20260925 13:12:21.098084 2036 ts_tablet_manager.cc:1939] T dfac8253f8d549b4b0c26704148821a8 P 4f3b20af407a4516930e0146b98b333c: Deleting consensus metadata
I20260925 13:12:21.098407 1493 catalog_manager.cc:4841] TS 4f3b20af407a4516930e0146b98b333c (127.0.98.1:46843): tablet dfac8253f8d549b4b0c26704148821a8 (table test-table [id=22a7cc876f084d6383631fac98c48742]) successfully deleted
Sep 25 13:12:21 dist-test-slave-2d0s krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:21.108541 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.097606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.0.98.1:46843)
0925 13:12:21.097651 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.097653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.098436 (+ 783us) server_negotiation.cc:407] Connection header received
0925 13:12:21.098536 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.098537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.098548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.098576 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:21.099335 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.100070 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.101136 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.101414 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.101487 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.105031 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:21.105042 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:21.105042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:21.105050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:21.107136 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:21.107553 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:21.107557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:21.107557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:21.107619 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:21.107809 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:21.107810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:21.107810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:21.107998 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:21.108122 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.108338 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.108425 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:12:21.109097 1699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44298
I20260925 13:12:21.118579 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:21.110465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.0.98.62:36463)
0925 13:12:21.110518 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:21.110520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:21.110625 (+ 105us) server_negotiation.cc:407] Connection header received
0925 13:12:21.110725 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:21.110726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:21.110739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:21.110769 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:21.111533 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.112382 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.113431 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:21.113870 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:21.113913 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:21.115546 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:21.115548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:21.115548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:21.115556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:21.117311 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:21.117763 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:21.117764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:21.117764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:21.117824 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:21.118043 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:21.118044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:21.118044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:21.118183 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:21.118288 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:21.118391 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 13:12:21.118479 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 13:12:21.118767 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45424
I20260925 13:12:21.119516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1579
I20260925 13:12:21.124843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1719
I20260925 13:12:21.129539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1859
I20260925 13:12:21.134157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 1475
2026-09-25T13:12:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2510 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:21 dist-test-slave-2d0s krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:21 dist-test-slave-2d0s krb5kdc[2082](info): set up 2 sockets
Sep 25 13:12:21 dist-test-slave-2d0s krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260925 13:12:22.161392 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:22.280385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:22.362465 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:22.362741 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:22.362779 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:22.364010 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:22.364090 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 13:12:22.364115 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:22.364141 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:22.364174 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:22.364203 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:22.365834 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:22.366125 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:22.366374 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:22.371414 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:22.371477 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:22.371582 2095 server_base.cc:1034] running on GCE node
W20260925 13:12:22.371429 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:22.371941 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:22.372253 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:22.373412 2095 hybrid_clock.cc:648] HybridClock initialized: now 1790341942373405 us; error 53 us; skew 500 ppm
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:22.376005 2095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:22.376528 2095 webserver.cc:460] Webserver started at http://127.0.98.62:33597/ using document root <none> and password file <none>
I20260925 13:12:22.376778 2095 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:22.376823 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:22.377002 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:22.378206 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "ef21f788b6554074aa6101e69498c062"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "9f269f075177fa771ad8782667227c08"
server_key_iv: "7f2b0b67c72b2433897af1eb2976d97a"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.378599 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef21f788b6554074aa6101e69498c062"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "9f269f075177fa771ad8782667227c08"
server_key_iv: "7f2b0b67c72b2433897af1eb2976d97a"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.380038 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 13:12:22.380790 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.380983 2095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:22.381055 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "ef21f788b6554074aa6101e69498c062"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "9f269f075177fa771ad8782667227c08"
server_key_iv: "7f2b0b67c72b2433897af1eb2976d97a"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.381147 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:22.412286 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:22.414379 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:22.414535 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:22.420163 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38221
I20260925 13:12:22.420257 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38221 every 8 connection(s)
I20260925 13:12:22.420616 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:22.421268 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:22.423700 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Bootstrap starting.
I20260925 13:12:22.424317 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:22.424625 2173 log.cc:824] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:22.425294 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: No bootstrap required, opened a new log
I20260925 13:12:22.425643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 2095
I20260925 13:12:22.425783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:22.425985 392 external_mini_cluster.cc:1428] Setting key b50cb52d7b5dd05d30f2520c4d085622
I20260925 13:12:22.427810 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef21f788b6554074aa6101e69498c062" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38221 } }
I20260925 13:12:22.427981 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:22.428025 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef21f788b6554074aa6101e69498c062, State: Initialized, Role: FOLLOWER
I20260925 13:12:22.428179 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef21f788b6554074aa6101e69498c062" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38221 } }
I20260925 13:12:22.428268 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:22.428311 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:22.428366 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:22.428915 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef21f788b6554074aa6101e69498c062" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38221 } }
I20260925 13:12:22.429032 2173 leader_election.cc:302] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef21f788b6554074aa6101e69498c062; no voters:
I20260925 13:12:22.429240 2173 leader_election.cc:288] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:22.429267 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:22.429435 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [term 1 LEADER]: Becoming Leader. State: Replica: ef21f788b6554074aa6101e69498c062, State: Running, Role: LEADER
I20260925 13:12:22.429531 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:22.429616 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef21f788b6554074aa6101e69498c062" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38221 } }
I20260925 13:12:22.430065 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef21f788b6554074aa6101e69498c062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef21f788b6554074aa6101e69498c062" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38221 } } }
I20260925 13:12:22.430156 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:22.430406 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef21f788b6554074aa6101e69498c062. Latest consensus state: current_term: 1 leader_uuid: "ef21f788b6554074aa6101e69498c062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef21f788b6554074aa6101e69498c062" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38221 } } }
I20260925 13:12:22.430531 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062 [sys.catalog]: This master's current role is: LEADER
W20260925 13:12:22.431306 2194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:12:22.431828 2194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:22.431901 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:22.432097 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:22.433614 2187 catalog_manager.cc:1347] Generated new cluster ID: 71a113cbef354d14a1e11252ac8fbf51
I20260925 13:12:22.433686 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:22.438035 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.426515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.0.98.62:38221)
0925 13:12:22.426699 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.426708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.426743 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:12:22.427789 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.427796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.427846 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.427986 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.428708 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.429943 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.431030 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.431451 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.431530 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.434346 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.434350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.434376 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.434389 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.436093 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.436475 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.436479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.436479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.436932 (+ 453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.437163 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.437164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.437164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.437395 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.437558 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.437651 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.437826 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260925 13:12:22.445024 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:22.445573 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:22.449041 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Generated new TSK 0
I20260925 13:12:22.449224 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:22.484803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38221
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:22.586409 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:22.586622 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:22.586648 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:22.587805 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:22.587849 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:22.587869 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 13:12:22.587881 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:22.587901 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:22.589365 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:22.589632 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:22.589884 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:22.594949 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:22.595041 2199 server_base.cc:1034] running on GCE node
W20260925 13:12:22.594976 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:22.595173 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:22.595403 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:22.595683 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:22.596849 2199 hybrid_clock.cc:648] HybridClock initialized: now 1790341942596832 us; error 39 us; skew 500 ppm
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:22.599656 2199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:22.600054 2199 webserver.cc:460] Webserver started at http://127.0.98.1:37537/ using document root <none> and password file <none>
I20260925 13:12:22.600301 2199 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:22.600378 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:22.600528 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:22.601577 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "257466f3e6134234bd1f0c947200cffa"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "1fb7f4ee63058925fe7192ceb169015e"
server_key_iv: "42e52ae177744dbdf086cbe371eff3e2"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.602031 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "257466f3e6134234bd1f0c947200cffa"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "1fb7f4ee63058925fe7192ceb169015e"
server_key_iv: "42e52ae177744dbdf086cbe371eff3e2"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.603334 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:22.604075 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.604266 2199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:22.604382 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "257466f3e6134234bd1f0c947200cffa"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "1fb7f4ee63058925fe7192ceb169015e"
server_key_iv: "42e52ae177744dbdf086cbe371eff3e2"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.604454 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:22.614028 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:22.616314 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:22.616483 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:22.616995 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:22.617063 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.617107 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:22.617144 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.624051 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45455
I20260925 13:12:22.624096 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45455 every 8 connection(s)
I20260925 13:12:22.624429 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:22.627617 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.617605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54087 (local address 127.0.98.62:38221)
0925 13:12:22.617692 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.617694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.617944 (+ 250us) server_negotiation.cc:407] Connection header received
0925 13:12:22.618675 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.618677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.618692 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.618791 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.619675 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.620537 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.622118 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.622464 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.622546 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.625235 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.625236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.625236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.625243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.626528 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.626881 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.626881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.626882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.626929 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.627139 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.627140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.627140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.627233 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.627280 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.627497 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.627556 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 13:12:22.627884 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.617561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.98.1:54087)
0925 13:12:22.617782 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.617829 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.618551 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.618880 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.618885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.619224 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:22.619560 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.619567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.620637 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.620638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.622022 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.622025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.622575 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.622576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.622779 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.623410 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:22.623420 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:22.625146 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:22.626598 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.626604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.626605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.626811 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.626985 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.626985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.626985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.627088 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.627282 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:22.627285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:22.627390 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.627477 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.627591 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":142,"threads_started":1}
I20260925 13:12:22.628427 2330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38221
I20260925 13:12:22.628508 2330 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:22.628688 2330 heartbeater.cc:511] Master 127.0.98.62:38221 requested a full tablet report, sending...
I20260925 13:12:22.629228 2127 ts_manager.cc:194] Registered new tserver with Master: 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455)
I20260925 13:12:22.630020 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54087
I20260925 13:12:22.630528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 2199
I20260925 13:12:22.630625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:22.630801 392 external_mini_cluster.cc:1428] Setting key 359ddec4492fa30fd45bb8e49b432b74
I20260925 13:12:22.635620 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.629360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36939 (local address 127.0.98.62:38221)
0925 13:12:22.629399 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.629401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.629610 (+ 209us) server_negotiation.cc:407] Connection header received
0925 13:12:22.629732 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.629733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.629742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.629784 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.630152 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.630868 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.631747 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.632063 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.632136 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.633080 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.633082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.633082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.633089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.634445 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.634861 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.634862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.634862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.634909 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.635113 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.635114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.635114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.635225 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.635300 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.635455 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.635530 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:12:22.635636 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.629405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.98.1:36939)
0925 13:12:22.629540 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.629549 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.629637 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.629796 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.629797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.629986 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:22.630034 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.630035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.630984 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.630985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.631672 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.631674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.632136 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.632137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.632189 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.632688 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:22.632694 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:22.632997 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:22.634526 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.634527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.634528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.634776 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.634978 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.634979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.634980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.635044 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.635292 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:22.635293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:22.635334 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.635431 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.635532 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:22.656335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38221
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:22.740195 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:22.740418 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:22.740443 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:22.741647 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:22.741711 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:22.741732 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 13:12:22.741787 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:22.741813 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:22.743239 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:22.743510 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:22.743742 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:22.748705 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:22.748714 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:22.748714 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:22.748790 2339 server_base.cc:1034] running on GCE node
I20260925 13:12:22.749185 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:22.749465 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:22.750615 2339 hybrid_clock.cc:648] HybridClock initialized: now 1790341942750589 us; error 40 us; skew 500 ppm
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:22.753304 2339 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:22.753888 2339 webserver.cc:460] Webserver started at http://127.0.98.2:43177/ using document root <none> and password file <none>
I20260925 13:12:22.754123 2339 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:22.754169 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:22.754330 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:22.755415 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "45d9a8c5c6a045c887d2f26a7d24ce25"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "6c776817cbdb4fa4f2d7af690b1370ec"
server_key_iv: "e727e4b02e6527af6c581e329e7cd95b"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.755779 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45d9a8c5c6a045c887d2f26a7d24ce25"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "6c776817cbdb4fa4f2d7af690b1370ec"
server_key_iv: "e727e4b02e6527af6c581e329e7cd95b"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.757113 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:22.757908 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.758070 2339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:22.758153 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "45d9a8c5c6a045c887d2f26a7d24ce25"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "6c776817cbdb4fa4f2d7af690b1370ec"
server_key_iv: "e727e4b02e6527af6c581e329e7cd95b"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.758206 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:22.777642 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:22.779785 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:22.779963 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:22.780530 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:22.780571 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.780596 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:22.780644 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.787359 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34441
I20260925 13:12:22.787454 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34441 every 8 connection(s)
I20260925 13:12:22.787760 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:22.791428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 2339
I20260925 13:12:22.791501 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.781130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33211 (local address 127.0.98.62:38221)
0925 13:12:22.781184 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.781186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.781457 (+ 271us) server_negotiation.cc:407] Connection header received
0925 13:12:22.782094 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.782095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.782108 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.782164 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.783035 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.783841 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.785439 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.785802 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.785909 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.788482 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.788483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.788483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.788490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.790126 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.790518 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.790519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.790519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.790569 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.790787 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.790788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.790788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.790897 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.790948 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.791304 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.791390 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:12:22.791519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:22.791693 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.781116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.98.2:33211)
0925 13:12:22.781315 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.781351 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.781986 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.782240 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.782245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.782657 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:22.782949 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.782956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.783939 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.783940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.785314 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.785317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.785910 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.785911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.786079 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.786735 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:22.786746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:22.788396 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:22.790212 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.790217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.790219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.790450 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.790634 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.790634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.790634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.790730 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.790985 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:22.790991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:22.791124 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.791240 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.791441 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260925 13:12:22.791710 392 external_mini_cluster.cc:1428] Setting key 465d423de1f1658ed8fd854321395ac6
I20260925 13:12:22.792218 2470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38221
I20260925 13:12:22.792321 2470 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:22.792536 2470 heartbeater.cc:511] Master 127.0.98.62:38221 requested a full tablet report, sending...
I20260925 13:12:22.792896 2124 ts_manager.cc:194] Registered new tserver with Master: 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441)
I20260925 13:12:22.793360 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33211
I20260925 13:12:22.799386 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.793472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58537 (local address 127.0.98.62:38221)
0925 13:12:22.793508 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.793510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.793518 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:22.793536 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.793537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.793547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.793600 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.794003 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.794508 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.795515 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.795717 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.795798 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.796779 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.796781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.796781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.796787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.798212 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.798574 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.798576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.798576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.798632 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.798840 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.798842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.798842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.798989 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.799063 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.799214 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.799289 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 13:12:22.799410 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.793262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.98.2:58537)
0925 13:12:22.793401 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.793411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.793473 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.793630 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.793631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.793851 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:22.793918 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.793920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.794612 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.794613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.795432 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.795435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.795802 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.795804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.795877 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.796397 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:22.796401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:22.796675 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:22.798308 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.798309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.798309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.798488 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.798697 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.798698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.798699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.798757 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.799064 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:22.799065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:22.799108 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.799192 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.799290 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:22.817811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38221
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:22.913203 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:22.913439 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:22.913481 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:22.914697 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:22.914759 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:22.914798 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 13:12:22.914831 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:22.914875 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:22.916355 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:22.916594 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:22.916808 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:22.921446 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:22.921486 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:22.921474 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:22.921774 2479 server_base.cc:1034] running on GCE node
I20260925 13:12:22.921967 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:22.922254 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:22.923405 2479 hybrid_clock.cc:648] HybridClock initialized: now 1790341942923389 us; error 38 us; skew 500 ppm
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:22.925889 2479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:22.926331 2479 webserver.cc:460] Webserver started at http://127.0.98.3:40629/ using document root <none> and password file <none>
I20260925 13:12:22.926523 2479 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:22.926564 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:22.926654 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:22.927696 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "db77501ecdc146ef8bb5f256293571c3"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "e0913fbb267c3f6057972ad584a95a44"
server_key_iv: "76ddfcb36376805e219c0adfb10ef893"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.928081 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "db77501ecdc146ef8bb5f256293571c3"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "e0913fbb267c3f6057972ad584a95a44"
server_key_iv: "76ddfcb36376805e219c0adfb10ef893"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.929479 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:22.930456 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.930649 2479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:22.930742 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "db77501ecdc146ef8bb5f256293571c3"
format_stamp: "Formatted at 2026-09-25 13:12:22 on dist-test-slave-2d0s"
server_key: "e0913fbb267c3f6057972ad584a95a44"
server_key_iv: "76ddfcb36376805e219c0adfb10ef893"
server_key_version: "encryptionkey@0"
I20260925 13:12:22.930827 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:22.943984 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:22.946236 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:22.946403 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:22.946933 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:22.946996 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.947031 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:22.947075 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:22.953325 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35763
I20260925 13:12:22.953410 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35763 every 8 connection(s)
I20260925 13:12:22.953814 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:22.957106 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.947453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51513 (local address 127.0.98.62:38221)
0925 13:12:22.947507 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.947510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.947699 (+ 189us) server_negotiation.cc:407] Connection header received
0925 13:12:22.948217 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.948218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.948230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.948287 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.949293 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.950186 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.951736 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.952031 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.952097 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.954792 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.954793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.954793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.954800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.955982 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.956344 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.956344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.956345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.956394 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.956610 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.956611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.956611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.956709 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.956760 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.956985 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.957045 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:12:22.957305 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.947404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.98.3:51513)
0925 13:12:22.947568 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.947607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.948115 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.948367 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.948371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.948768 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:22.949160 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.949167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.950274 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.950275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.951634 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.951637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.952104 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.952105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.952249 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.952869 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:22.952879 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:22.954708 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:22.956055 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.956060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.956062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.956276 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.956463 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.956463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.956463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.956559 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.956762 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:22.956766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:22.956893 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.956973 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.957091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260925 13:12:22.957813 2610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38221
I20260925 13:12:22.957917 2610 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:22.958094 2610 heartbeater.cc:511] Master 127.0.98.62:38221 requested a full tablet report, sending...
I20260925 13:12:22.958482 2126 ts_manager.cc:194] Registered new tserver with Master: db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763)
I20260925 13:12:22.958957 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51513
I20260925 13:12:22.962867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 2479
I20260925 13:12:22.962966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:22.963152 392 external_mini_cluster.cc:1428] Setting key cabb15910c56154a7dbd00ffae83706e
I20260925 13:12:22.964082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:12:22.964823 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.958845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52723 (local address 127.0.98.62:38221)
0925 13:12:22.958894 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.958896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.959199 (+ 303us) server_negotiation.cc:407] Connection header received
0925 13:12:22.959259 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.959260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.959270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.959318 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.959636 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.960082 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.960817 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.961065 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.961131 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.962105 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.962106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.962106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.962112 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.963733 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.964058 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.964059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.964060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.964115 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.964319 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.964319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.964320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.964433 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.964508 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.964654 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.964730 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:22.964836 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.958782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.98.3:52723)
0925 13:12:22.959132 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.959142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.959208 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.959321 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.959322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.959516 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:22.959569 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.959571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.960164 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.960164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.960723 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.960725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.961136 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.961137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.961188 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.961646 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:22.961649 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:22.962036 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:22.963792 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.963793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.963794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.963982 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.964189 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:22.964190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:22.964191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:22.964249 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:22.964499 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:22.964500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:22.964542 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.964631 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.964733 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":90,"threads_started":1}
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:12:22 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:22.987618 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.980250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.0.98.62:38221)
0925 13:12:22.980339 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.980341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.980355 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:12:22.980381 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.980382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.980393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.980462 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:22.981065 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.981559 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.982350 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.982543 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.982620 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.984974 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:22.984975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:22.984976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:22.984985 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:22.986457 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.986821 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.986822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.986823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.986888 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:22.987084 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:22.987084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:22.987085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:22.987199 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:22.987307 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.987426 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.987510 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 13:12:22.989362 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:22.991009 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:22.997443 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.993427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.0.98.2:34441)
0925 13:12:22.994046 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.994050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.994061 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:22.994116 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.994118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.994133 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.994339 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:22.994399 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.995451 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.996519 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.997018 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.997104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.997167 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.997309 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.997350 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":73,"threads_started":1}
I20260925 13:12:22.997435 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.993324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.0.1:55844)
0925 13:12:22.993470 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.993509 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.993574 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.994235 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.994236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.994239 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:22.994274 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.994275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.995534 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.995535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.996436 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.996438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.997103 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.997104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.997178 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.997191 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.997291 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.997344 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1,"thread_start_us":44,"threads_started":1}
I20260925 13:12:22.997632 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.993521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45455 (local address 127.0.0.1:44626)
0925 13:12:22.993715 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.993723 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.993762 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.994182 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.994183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.994185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:22.994250 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.994251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.995135 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.995136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.996449 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.996451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.997387 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.997388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.997465 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.997470 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.997502 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.997553 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260925 13:12:22.997850 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.993549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.98.3:35763)
0925 13:12:22.993757 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.993763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.993772 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:22.993808 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.993810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.993824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.993928 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:22.994181 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.995323 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.996560 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.997287 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.997359 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.997411 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.997712 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.997760 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260925 13:12:22.997637 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.993339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35763 (local address 127.0.0.1:52908)
0925 13:12:22.993631 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:22.993641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:22.993712 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:22.993941 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:22.993944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:22.993946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:22.994094 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.994096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.995419 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.995420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.996475 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:22.996476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.997439 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.997440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:22.997504 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.997507 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:22.997550 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:12:22.997579 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260925 13:12:22.998021 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:22.993807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.0.98.1:45455)
0925 13:12:22.993975 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:22.993979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:22.993990 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:22.994049 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:22.994051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:22.994085 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:22.994326 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:22.994385 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.995049 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.996556 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:22.997287 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:22.997762 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:22.997835 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:22.997862 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:12:22.997908 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260925 13:12:22.998430 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 (DEFAULT_TABLE table=test-table [id=57be08df6c214baf8a2005d761920d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:22.998605 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 (DEFAULT_TABLE table=test-table [id=57be08df6c214baf8a2005d761920d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:22.998603 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 (DEFAULT_TABLE table=test-table [id=57be08df6c214baf8a2005d761920d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:22.998847 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:22.998936 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:22.998944 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.002254 2633 tablet_bootstrap.cc:492] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Bootstrap starting.
I20260925 13:12:23.002379 2634 tablet_bootstrap.cc:492] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Bootstrap starting.
I20260925 13:12:23.002882 2633 tablet_bootstrap.cc:654] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.003187 2633 log.cc:824] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:23.003227 2634 tablet_bootstrap.cc:654] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.003564 2634 log.cc:824] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:23.003921 2633 tablet_bootstrap.cc:492] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: No bootstrap required, opened a new log
I20260925 13:12:23.003906 2635 tablet_bootstrap.cc:492] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Bootstrap starting.
I20260925 13:12:23.003996 2633 ts_tablet_manager.cc:1397] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:23.004274 2634 tablet_bootstrap.cc:492] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: No bootstrap required, opened a new log
I20260925 13:12:23.004338 2634 ts_tablet_manager.cc:1397] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:12:23.004516 2635 tablet_bootstrap.cc:654] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.004822 2635 log.cc:824] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:23.005611 2635 tablet_bootstrap.cc:492] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: No bootstrap required, opened a new log
I20260925 13:12:23.005720 2635 ts_tablet_manager.cc:1397] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:12:23.006091 2633 raft_consensus.cc:348] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.006243 2633 raft_consensus.cc:374] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.006284 2633 raft_consensus.cc:729] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.006413 2633 consensus_queue.cc:260] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.006459 2634 raft_consensus.cc:348] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.006673 2634 raft_consensus.cc:374] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.006708 2634 raft_consensus.cc:729] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.006731 2633 ts_tablet_manager.cc:1428] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:23.006820 2634 consensus_queue.cc:260] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.006902 2470 heartbeater.cc:503] Master 127.0.98.62:38221 was elected leader, sending a full tablet report...
I20260925 13:12:23.007061 2634 ts_tablet_manager.cc:1428] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:23.007107 2610 heartbeater.cc:503] Master 127.0.98.62:38221 was elected leader, sending a full tablet report...
I20260925 13:12:23.007754 2635 raft_consensus.cc:348] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.007937 2635 raft_consensus.cc:374] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.007983 2635 raft_consensus.cc:729] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257466f3e6134234bd1f0c947200cffa, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.008085 2640 raft_consensus.cc:482] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:23.008121 2635 consensus_queue.cc:260] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.008160 2640 raft_consensus.cc:504] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.008383 2635 ts_tablet_manager.cc:1428] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:23.008468 2640 leader_election.cc:288] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441), 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455)
I20260925 13:12:23.008522 2330 heartbeater.cc:503] Master 127.0.98.62:38221 was elected leader, sending a full tablet report...
I20260925 13:12:23.011550 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.009040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49975 (local address 127.0.98.1:45455)
0925 13:12:23.009077 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.009078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.009087 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:23.009203 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.009204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.009214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.009261 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:23.009413 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.009954 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.010837 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.011196 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.011285 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.011295 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.011446 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.011484 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:23.011551 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.009039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39863 (local address 127.0.98.2:34441)
0925 13:12:23.009103 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.009104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.009112 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:23.009132 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.009133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.009142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.009185 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:23.009336 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.009896 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.010805 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.011191 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.011284 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.011295 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.011452 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.011486 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:12:23.011572 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.008613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.3:39863)
0925 13:12:23.008868 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.008879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.008919 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.009201 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.009202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.009203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:23.009265 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.009267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.009972 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.009973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.010718 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.010720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.011284 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.011285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.011359 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.011363 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.011445 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.011486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260925 13:12:23.011583 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.008705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45455 (local address 127.0.98.3:49975)
0925 13:12:23.008936 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.008947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.008992 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.009263 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.009263 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.009265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:23.009336 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.009337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.010028 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.010030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.010742 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.010744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.011289 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.011290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.011367 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.011375 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.011440 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.011481 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260925 13:12:23.012118 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e705dd76b1b43b5a4fb2f31cfdf4464" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa" is_pre_election: true
I20260925 13:12:23.012121 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e705dd76b1b43b5a4fb2f31cfdf4464" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" is_pre_election: true
I20260925 13:12:23.012225 2284 raft_consensus.cc:2454] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 0.
I20260925 13:12:23.012252 2424 raft_consensus.cc:2454] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 0.
I20260925 13:12:23.012463 2495 leader_election.cc:302] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d9a8c5c6a045c887d2f26a7d24ce25, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:23.012573 2640 raft_consensus.cc:2781] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:23.012624 2640 raft_consensus.cc:482] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:23.012653 2640 raft_consensus.cc:3037] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.013291 2640 raft_consensus.cc:504] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.013417 2640 leader_election.cc:288] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 election: Requested vote from peers 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441), 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455)
I20260925 13:12:23.013585 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e705dd76b1b43b5a4fb2f31cfdf4464" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25"
I20260925 13:12:23.013613 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e705dd76b1b43b5a4fb2f31cfdf4464" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa"
I20260925 13:12:23.013695 2424 raft_consensus.cc:3037] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.013708 2284 raft_consensus.cc:3037] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.014564 2284 raft_consensus.cc:2454] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 1.
I20260925 13:12:23.014564 2424 raft_consensus.cc:2454] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 1.
I20260925 13:12:23.014720 2495 leader_election.cc:302] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257466f3e6134234bd1f0c947200cffa, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:23.014817 2640 raft_consensus.cc:2781] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:23.014951 2640 raft_consensus.cc:686] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 1 LEADER]: Becoming Leader. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Running, Role: LEADER
I20260925 13:12:23.015039 2640 consensus_queue.cc:237] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.015785 2126 catalog_manager.cc:5495] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db77501ecdc146ef8bb5f256293571c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "db77501ecdc146ef8bb5f256293571c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } health_report { overall_health: HEALTHY } } }
W20260925 13:12:23.038445 2471 tablet.cc:2369] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 13:12:23 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:23.065162 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.058536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.0.98.62:38221)
0925 13:12:23.058606 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.058608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.058716 (+ 108us) server_negotiation.cc:407] Connection header received
0925 13:12:23.058761 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.058762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.058771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.058824 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:23.059301 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.059782 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.060533 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.060710 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.060758 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.062772 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:23.062773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:23.062773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:23.062779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:23.064254 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:23.064584 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:23.064584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:23.064585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:23.064639 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:23.064781 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:23.064782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:23.064782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:23.064869 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:23.064921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.065036 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.065099 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:12:23.065276 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.058480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38221 (local address 127.0.0.1:34090)
0925 13:12:23.058642 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.058652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.058719 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.058854 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.058855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.059187 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:23.059239 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.059241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.059848 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.059848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.060418 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.060420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.060770 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.060771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.060809 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.061304 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:23.061308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:23.062700 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:23.064333 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:23.064338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:23.064340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:23.064524 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:23.064694 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:23.064694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:23.064695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:23.064746 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:23.064930 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:23.064935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:23.064973 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.065022 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.065145 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":41,"thread_start_us":108,"threads_started":1}
I20260925 13:12:23.067176 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34090:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 13:12:23.067351 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:23.068744 2264 tablet_service.cc:1467] Processing CreateTablet for tablet b22d441a88d84203bccabc544cd0ca50 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1feb814098bb41c691bf8a7f09362991]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:23.068742 2544 tablet_service.cc:1467] Processing CreateTablet for tablet b22d441a88d84203bccabc544cd0ca50 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1feb814098bb41c691bf8a7f09362991]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:23.068764 2397 tablet_service.cc:1467] Processing CreateTablet for tablet b22d441a88d84203bccabc544cd0ca50 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1feb814098bb41c691bf8a7f09362991]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:23.068907 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22d441a88d84203bccabc544cd0ca50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.068907 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22d441a88d84203bccabc544cd0ca50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.068922 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22d441a88d84203bccabc544cd0ca50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.070246 2633 tablet_bootstrap.cc:492] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: Bootstrap starting.
I20260925 13:12:23.070564 2635 tablet_bootstrap.cc:492] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: Bootstrap starting.
I20260925 13:12:23.070858 2634 tablet_bootstrap.cc:492] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: Bootstrap starting.
I20260925 13:12:23.071002 2633 tablet_bootstrap.cc:654] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.071271 2634 tablet_bootstrap.cc:654] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.071346 2635 tablet_bootstrap.cc:654] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.071820 2633 tablet_bootstrap.cc:492] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: No bootstrap required, opened a new log
I20260925 13:12:23.071849 2634 tablet_bootstrap.cc:492] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: No bootstrap required, opened a new log
I20260925 13:12:23.071871 2633 ts_tablet_manager.cc:1397] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:23.071894 2634 ts_tablet_manager.cc:1397] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:23.072000 2635 tablet_bootstrap.cc:492] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: No bootstrap required, opened a new log
I20260925 13:12:23.072048 2635 ts_tablet_manager.cc:1397] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:23.072024 2633 raft_consensus.cc:348] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.072034 2634 raft_consensus.cc:348] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.072098 2634 raft_consensus.cc:374] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.072094 2633 raft_consensus.cc:374] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.072116 2634 raft_consensus.cc:729] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.072124 2633 raft_consensus.cc:729] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.072156 2634 consensus_queue.cc:260] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.072170 2633 consensus_queue.cc:260] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.072192 2635 raft_consensus.cc:348] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.072260 2634 ts_tablet_manager.cc:1428] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:23.072268 2635 raft_consensus.cc:374] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.072268 2633 ts_tablet_manager.cc:1428] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:23.072310 2635 raft_consensus.cc:729] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257466f3e6134234bd1f0c947200cffa, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.072361 2635 consensus_queue.cc:260] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.072391 2640 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.072464 2635 ts_tablet_manager.cc:1428] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:23.072463 2640 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:23.072458 2639 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.072538 2639 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:23.072633 2641 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.072741 2641 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 13:12:23.125205 2331 tablet.cc:2369] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:23.162459 2639 raft_consensus.cc:482] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:23.162585 2639 raft_consensus.cc:504] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.162925 2639 leader_election.cc:288] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763)
I20260925 13:12:23.166813 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.163176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50687 (local address 127.0.98.3:35763)
0925 13:12:23.163216 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.163218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.163373 (+ 155us) server_negotiation.cc:407] Connection header received
0925 13:12:23.163406 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.163407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.163416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.163460 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:23.163621 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.164301 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.165897 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.166447 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.166525 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.166535 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.166703 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.166744 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:23.166813 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.163151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53905 (local address 127.0.98.1:45455)
0925 13:12:23.163200 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.163203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.163212 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:23.163240 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.163241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.163250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.163312 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:23.163525 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.164297 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.165896 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.166461 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.166525 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.166535 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.166707 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.166744 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:12:23.166846 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.163064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45455 (local address 127.0.98.2:53905)
0925 13:12:23.163125 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.163138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.163184 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.163346 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.163347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.163349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:23.163433 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.163435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.164382 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.164382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.165806 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.165809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.166534 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.166536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.166623 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.166626 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.166683 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.166723 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":87}
I20260925 13:12:23.166854 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.163117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35763 (local address 127.0.98.2:50687)
0925 13:12:23.163268 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.163284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.163351 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.163479 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.163479 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.163481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:23.163555 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.163556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.164378 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.164378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.165806 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.165809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.166558 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.166559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.166623 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.166626 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.166683 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.166723 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":108,"thread_start_us":107,"threads_started":1}
I20260925 13:12:23.167166 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22d441a88d84203bccabc544cd0ca50" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa" is_pre_election: true
I20260925 13:12:23.167253 2284 raft_consensus.cc:2454] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 0.
I20260925 13:12:23.167335 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22d441a88d84203bccabc544cd0ca50" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db77501ecdc146ef8bb5f256293571c3" is_pre_election: true
I20260925 13:12:23.167462 2549 raft_consensus.cc:2454] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 0.
I20260925 13:12:23.167446 2355 leader_election.cc:302] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257466f3e6134234bd1f0c947200cffa, 45d9a8c5c6a045c887d2f26a7d24ce25; no voters:
I20260925 13:12:23.167586 2639 raft_consensus.cc:2781] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:23.167634 2639 raft_consensus.cc:482] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:23.167654 2639 raft_consensus.cc:3037] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.168381 2639 raft_consensus.cc:504] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.168552 2639 leader_election.cc:288] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 election: Requested vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763)
I20260925 13:12:23.168691 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22d441a88d84203bccabc544cd0ca50" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa"
I20260925 13:12:23.168773 2284 raft_consensus.cc:3037] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.168717 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22d441a88d84203bccabc544cd0ca50" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db77501ecdc146ef8bb5f256293571c3"
I20260925 13:12:23.168807 2549 raft_consensus.cc:3037] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.169401 2284 raft_consensus.cc:2454] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 1.
I20260925 13:12:23.169430 2549 raft_consensus.cc:2454] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 1.
I20260925 13:12:23.169570 2355 leader_election.cc:302] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257466f3e6134234bd1f0c947200cffa, 45d9a8c5c6a045c887d2f26a7d24ce25; no voters:
I20260925 13:12:23.169651 2639 raft_consensus.cc:2781] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:23.169828 2639 raft_consensus.cc:686] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 LEADER]: Becoming Leader. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Running, Role: LEADER
I20260925 13:12:23.169950 2639 consensus_queue.cc:237] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.170467 2655 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: TxnStatusTablet state changed. Reason: New leader 45d9a8c5c6a045c887d2f26a7d24ce25. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.170548 2655 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:23.170811 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 13:12:23.170846 2126 catalog_manager.cc:5495] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:23.170933 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 13:12:23.190600 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.187738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.0.1:55852)
0925 13:12:23.187833 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.187846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.187908 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.188122 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.188124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.188125 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:23.188184 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.188188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.188859 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.188860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.189491 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.189494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.189987 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.189988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.190063 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.190086 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.190398 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.190409 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.190464 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.190522 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 13:12:23.190608 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.187796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.0.98.2:34441)
0925 13:12:23.187859 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.187861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.187888 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:12:23.187990 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.187991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.188014 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.188114 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:23.188263 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.188755 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.189599 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.189891 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.189984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.190204 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.190322 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.190426 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.190471 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.190498 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:12:23.192926 2284 raft_consensus.cc:1261] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Refusing update from remote peer 45d9a8c5c6a045c887d2f26a7d24ce25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:23.192931 2549 raft_consensus.cc:1261] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Refusing update from remote peer 45d9a8c5c6a045c887d2f26a7d24ce25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:23.193250 2655 consensus_queue.cc:1035] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:23.193391 2639 consensus_queue.cc:1035] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:23.194557 2640 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: TxnStatusTablet state changed. Reason: New leader 45d9a8c5c6a045c887d2f26a7d24ce25. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.194649 2640 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:23.194792 2641 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: TxnStatusTablet state changed. Reason: New leader 45d9a8c5c6a045c887d2f26a7d24ce25. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.194880 2641 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:23.195528 2660 mvcc.cc:205] Tried to move back new op lower bound from 7333240599315177472 to 7333240599224844288. Current Snapshot: MvccSnapshot[applied={T|T < 7333240599315177472}]
I20260925 13:12:23.195760 2639 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.195851 2639 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:23.196115 2641 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.196198 2641 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 257466f3e6134234bd1f0c947200cffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:23.196625 2639 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.196704 2639 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P 45d9a8c5c6a045c887d2f26a7d24ce25: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:23.196929 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333240599315177472 to 7333240599224844288. Current Snapshot: MvccSnapshot[applied={T|T < 7333240599315177472}]
I20260925 13:12:23.197006 2640 tablet_replica.cc:439] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } } }
I20260925 13:12:23.197091 2640 tablet_replica.cc:442] T b22d441a88d84203bccabc544cd0ca50 P db77501ecdc146ef8bb5f256293571c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:23.200763 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.198343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.0.98.3:35763)
0925 13:12:23.198391 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.198393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.198402 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:23.198474 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.198475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.198491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.198558 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:23.198718 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.199201 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.199985 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.200188 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.200261 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.200441 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.200545 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.200638 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.200658 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.200683 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:12:23.204355 2611 tablet.cc:2369] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:23.205765 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.203251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.3:54013)
0925 13:12:23.203323 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.203335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.203387 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.203501 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.203501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.203502 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:23.203560 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.203561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.204190 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.204191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.204924 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.204926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.205295 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.205296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.205358 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.205382 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.205586 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.205589 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.205641 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.205703 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 13:12:23.205765 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.203319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54013 (local address 127.0.98.2:34441)
0925 13:12:23.203364 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.203367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.203375 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:23.203440 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.203441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.203453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.203501 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:23.203641 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.204103 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.205013 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.205212 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.205294 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.205486 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.205525 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.205585 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.205646 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.205689 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:23.208353 2284 raft_consensus.cc:1261] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Refusing update from remote peer db77501ecdc146ef8bb5f256293571c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:23.208459 2424 raft_consensus.cc:1261] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Refusing update from remote peer db77501ecdc146ef8bb5f256293571c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:23.208596 2640 consensus_queue.cc:1035] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:23.208727 2643 consensus_queue.cc:1035] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:23.210477 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333240599379537920 to 7333240598590205952. Current Snapshot: MvccSnapshot[applied={T|T < 7333240599379537920}]
I20260925 13:12:23.217243 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.214691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37239 (local address 127.0.98.3:35763)
0925 13:12:23.214733 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:23.214735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:23.214743 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:23.214816 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:23.214817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:23.214828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:23.214889 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:23.215039 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.215506 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.216511 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.216744 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.216826 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.217009 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.217043 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.217095 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:23.217157 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 13:12:23.217175 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:23.217262 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:23.214650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35763 (local address 127.0.98.2:37239)
0925 13:12:23.214704 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:23.214716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:23.214757 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:23.214892 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:23.214893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:23.214894 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:23.214958 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.214959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.215585 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.215586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.216404 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:23.216407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:23.216828 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:23.216828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:23.216900 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:23.216919 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:23.217101 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:23.217103 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:12:23.217152 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:12:23.217195 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 13:12:23.226605 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:23.226725 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:23.227494 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Sending DeleteTablet for 3 replicas of tablet 4e705dd76b1b43b5a4fb2f31cfdf4464
I20260925 13:12:23.227828 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55844
I20260925 13:12:23.227912 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44626
I20260925 13:12:23.228120 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52908
I20260925 13:12:23.228329 2680 tablet_replica.cc:330] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: stopping tablet replica
I20260925 13:12:23.228470 2680 raft_consensus.cc:2229] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:23.228611 2680 raft_consensus.cc:2258] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:23.228576 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:23.228713 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:23.229255 2680 ts_tablet_manager.cc:1905] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:23.229957 2397 tablet_service.cc:1467] Processing CreateTablet for tablet c0fe942289274d7cbd3f204194e314f2 (DEFAULT_TABLE table=test-table [id=d96a2954cd214417848a4b6bf4eb4170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:23.230104 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0fe942289274d7cbd3f204194e314f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.231359 2680 ts_tablet_manager.cc:1918] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:23.231392 2633 tablet_bootstrap.cc:492] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Bootstrap starting.
I20260925 13:12:23.231516 2680 log.cc:1196] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/4e705dd76b1b43b5a4fb2f31cfdf4464
I20260925 13:12:23.231652 2544 tablet_service.cc:1467] Processing CreateTablet for tablet c0fe942289274d7cbd3f204194e314f2 (DEFAULT_TABLE table=test-table [id=d96a2954cd214417848a4b6bf4eb4170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:23.231758 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0fe942289274d7cbd3f204194e314f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.231829 2633 tablet_bootstrap.cc:654] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.231998 2680 ts_tablet_manager.cc:1939] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P db77501ecdc146ef8bb5f256293571c3: Deleting consensus metadata
I20260925 13:12:23.232429 2112 catalog_manager.cc:4841] TS db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763): tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 (table test-table [id=57be08df6c214baf8a2005d761920d46]) successfully deleted
I20260925 13:12:23.232951 2634 tablet_bootstrap.cc:492] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Bootstrap starting.
I20260925 13:12:23.233096 2264 tablet_service.cc:1467] Processing CreateTablet for tablet c0fe942289274d7cbd3f204194e314f2 (DEFAULT_TABLE table=test-table [id=d96a2954cd214417848a4b6bf4eb4170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:23.233206 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0fe942289274d7cbd3f204194e314f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:23.233364 2634 tablet_bootstrap.cc:654] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.234050 2634 tablet_bootstrap.cc:492] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: No bootstrap required, opened a new log
I20260925 13:12:23.234094 2634 ts_tablet_manager.cc:1397] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:23.234220 2634 raft_consensus.cc:348] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.234292 2634 raft_consensus.cc:374] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.234313 2634 raft_consensus.cc:729] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.234352 2634 consensus_queue.cc:260] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.234450 2634 ts_tablet_manager.cc:1428] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:23.234516 2635 tablet_bootstrap.cc:492] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Bootstrap starting.
I20260925 13:12:23.234885 2635 tablet_bootstrap.cc:654] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:23.235683 2635 tablet_bootstrap.cc:492] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: No bootstrap required, opened a new log
I20260925 13:12:23.235735 2635 ts_tablet_manager.cc:1397] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:23.235868 2635 raft_consensus.cc:348] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.235939 2635 raft_consensus.cc:374] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.235958 2635 raft_consensus.cc:729] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257466f3e6134234bd1f0c947200cffa, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.236003 2635 consensus_queue.cc:260] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.236094 2635 ts_tablet_manager.cc:1428] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:23.236783 2679 tablet_replica.cc:330] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: stopping tablet replica
I20260925 13:12:23.236907 2679 raft_consensus.cc:2229] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:23.236960 2679 raft_consensus.cc:2258] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:23.237120 2678 tablet_replica.cc:330] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: stopping tablet replica
I20260925 13:12:23.237226 2678 raft_consensus.cc:2229] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:23.237272 2678 raft_consensus.cc:2258] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:23.237759 2678 ts_tablet_manager.cc:1905] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:23.237777 2679 ts_tablet_manager.cc:1905] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:23.239460 2679 ts_tablet_manager.cc:1918] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:23.239535 2679 log.cc:1196] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/4e705dd76b1b43b5a4fb2f31cfdf4464
I20260925 13:12:23.239795 2679 ts_tablet_manager.cc:1939] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 257466f3e6134234bd1f0c947200cffa: Deleting consensus metadata
I20260925 13:12:23.240118 2678 ts_tablet_manager.cc:1918] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:23.240183 2111 catalog_manager.cc:4841] TS 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455): tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 (table test-table [id=57be08df6c214baf8a2005d761920d46]) successfully deleted
I20260925 13:12:23.240223 2678 log.cc:1196] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/4e705dd76b1b43b5a4fb2f31cfdf4464
I20260925 13:12:23.240589 2678 ts_tablet_manager.cc:1939] T 4e705dd76b1b43b5a4fb2f31cfdf4464 P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting consensus metadata
I20260925 13:12:23.241017 2111 catalog_manager.cc:4841] TS 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441): tablet 4e705dd76b1b43b5a4fb2f31cfdf4464 (table test-table [id=57be08df6c214baf8a2005d761920d46]) successfully deleted
I20260925 13:12:23.242095 2633 tablet_bootstrap.cc:492] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: No bootstrap required, opened a new log
I20260925 13:12:23.242142 2633 ts_tablet_manager.cc:1397] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260925 13:12:23.242280 2633 raft_consensus.cc:348] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.242345 2633 raft_consensus.cc:374] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:23.242365 2633 raft_consensus.cc:729] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Initialized, Role: FOLLOWER
I20260925 13:12:23.242424 2633 consensus_queue.cc:260] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.242507 2633 ts_tablet_manager.cc:1428] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:23.368124 2643 raft_consensus.cc:482] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:23.368254 2643 raft_consensus.cc:504] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.368458 2643 leader_election.cc:288] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441)
I20260925 13:12:23.368847 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0fe942289274d7cbd3f204194e314f2" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa" is_pre_election: true
I20260925 13:12:23.368866 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0fe942289274d7cbd3f204194e314f2" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" is_pre_election: true
I20260925 13:12:23.368959 2284 raft_consensus.cc:2454] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 0.
I20260925 13:12:23.368975 2424 raft_consensus.cc:2454] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 0.
I20260925 13:12:23.369154 2495 leader_election.cc:302] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257466f3e6134234bd1f0c947200cffa, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:23.369282 2643 raft_consensus.cc:2781] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:23.369333 2643 raft_consensus.cc:482] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:23.369365 2643 raft_consensus.cc:3037] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.370136 2643 raft_consensus.cc:504] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.370282 2643 leader_election.cc:288] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 election: Requested vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441)
I20260925 13:12:23.370541 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0fe942289274d7cbd3f204194e314f2" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa"
I20260925 13:12:23.370565 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0fe942289274d7cbd3f204194e314f2" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25"
I20260925 13:12:23.370636 2284 raft_consensus.cc:3037] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.370636 2424 raft_consensus.cc:3037] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:23.371419 2424 raft_consensus.cc:2454] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 1.
I20260925 13:12:23.371446 2284 raft_consensus.cc:2454] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 1.
I20260925 13:12:23.371645 2495 leader_election.cc:302] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d9a8c5c6a045c887d2f26a7d24ce25, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:23.371740 2643 raft_consensus.cc:2781] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:23.371806 2643 raft_consensus.cc:686] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 1 LEADER]: Becoming Leader. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Running, Role: LEADER
I20260925 13:12:23.371863 2643 consensus_queue.cc:237] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:23.372470 2124 catalog_manager.cc:5495] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db77501ecdc146ef8bb5f256293571c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "db77501ecdc146ef8bb5f256293571c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } health_report { overall_health: HEALTHY } } }
I20260925 13:12:23.388052 2424 raft_consensus.cc:1261] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Refusing update from remote peer db77501ecdc146ef8bb5f256293571c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:23.388104 2284 raft_consensus.cc:1261] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Refusing update from remote peer db77501ecdc146ef8bb5f256293571c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:23.388326 2643 consensus_queue.cc:1035] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:23.388465 2640 consensus_queue.cc:1035] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:12:23.392788 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 13:12:24.914866 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:24.914986 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:24.915918 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Sending DeleteTablet for 3 replicas of tablet c0fe942289274d7cbd3f204194e314f2
I20260925 13:12:24.916249 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet c0fe942289274d7cbd3f204194e314f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55844
I20260925 13:12:24.916280 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet c0fe942289274d7cbd3f204194e314f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44626
I20260925 13:12:24.916319 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet c0fe942289274d7cbd3f204194e314f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52908
I20260925 13:12:24.916482 2710 tablet_replica.cc:330] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: stopping tablet replica
I20260925 13:12:24.916538 2710 raft_consensus.cc:2229] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:24.916610 2710 raft_consensus.cc:2258] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:24.916746 2711 tablet_replica.cc:330] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: stopping tablet replica
I20260925 13:12:24.916811 2711 raft_consensus.cc:2229] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:24.916874 2711 raft_consensus.cc:2258] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:24.916894 2710 ts_tablet_manager.cc:1905] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:24.917125 2711 ts_tablet_manager.cc:1905] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:24.918785 2711 ts_tablet_manager.cc:1918] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:12:24.918882 2711 log.cc:1196] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/c0fe942289274d7cbd3f204194e314f2
I20260925 13:12:24.919109 2711 ts_tablet_manager.cc:1939] T c0fe942289274d7cbd3f204194e314f2 P 257466f3e6134234bd1f0c947200cffa: Deleting consensus metadata
I20260925 13:12:24.919406 2710 ts_tablet_manager.cc:1918] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:12:24.919477 2710 log.cc:1196] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/c0fe942289274d7cbd3f204194e314f2
I20260925 13:12:24.919718 2111 catalog_manager.cc:4841] TS 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455): tablet c0fe942289274d7cbd3f204194e314f2 (table test-table [id=d96a2954cd214417848a4b6bf4eb4170]) successfully deleted
I20260925 13:12:24.919744 2710 ts_tablet_manager.cc:1939] T c0fe942289274d7cbd3f204194e314f2 P db77501ecdc146ef8bb5f256293571c3: Deleting consensus metadata
I20260925 13:12:24.919839 2709 tablet_replica.cc:330] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: stopping tablet replica
I20260925 13:12:24.919911 2709 raft_consensus.cc:2229] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:24.919955 2709 raft_consensus.cc:2258] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:24.920071 2112 catalog_manager.cc:4841] TS db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763): tablet c0fe942289274d7cbd3f204194e314f2 (table test-table [id=d96a2954cd214417848a4b6bf4eb4170]) successfully deleted
I20260925 13:12:24.920346 2709 ts_tablet_manager.cc:1905] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:24.921525 2709 ts_tablet_manager.cc:1918] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:12:24.921624 2709 log.cc:1196] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/c0fe942289274d7cbd3f204194e314f2
I20260925 13:12:24.921926 2709 ts_tablet_manager.cc:1939] T c0fe942289274d7cbd3f204194e314f2 P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting consensus metadata
I20260925 13:12:24.922286 2111 catalog_manager.cc:4841] TS 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441): tablet c0fe942289274d7cbd3f204194e314f2 (table test-table [id=d96a2954cd214417848a4b6bf4eb4170]) successfully deleted
Sep 25 13:12:24 dist-test-slave-2d0s krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:12:24 dist-test-slave-2d0s krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:24.939463 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:24.932632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.0.98.62:38221)
0925 13:12:24.932786 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:24.932788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:24.932798 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:24.932824 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:24.932825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:24.932841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:24.932910 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:24.933488 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:24.934053 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:24.934820 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:24.935015 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:24.935092 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:24.937300 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:24.937301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:24.937301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:24.937307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:24.938580 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:24.938867 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:24.938868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:24.938869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:24.938918 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:24.939072 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:24.939073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:24.939073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:24.939166 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:24.939236 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:24.939319 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 13:12:24.939375 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260925 13:12:24.941015 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:24.941157 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:24.942469 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab27bf1cb547e493a11a89f7aff22f (DEFAULT_TABLE table=test-table [id=37d91a2833f04ed394fa4754e0388fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:24.942577 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab27bf1cb547e493a11a89f7aff22f (DEFAULT_TABLE table=test-table [id=37d91a2833f04ed394fa4754e0388fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:24.942606 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab27bf1cb547e493a11a89f7aff22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:24.942592 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab27bf1cb547e493a11a89f7aff22f (DEFAULT_TABLE table=test-table [id=37d91a2833f04ed394fa4754e0388fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:24.942692 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab27bf1cb547e493a11a89f7aff22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:24.942732 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab27bf1cb547e493a11a89f7aff22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:24.943724 2720 tablet_bootstrap.cc:492] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Bootstrap starting.
I20260925 13:12:24.943953 2722 tablet_bootstrap.cc:492] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Bootstrap starting.
I20260925 13:12:24.944121 2720 tablet_bootstrap.cc:654] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:24.944286 2722 tablet_bootstrap.cc:654] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:24.944621 2721 tablet_bootstrap.cc:492] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Bootstrap starting.
I20260925 13:12:24.944720 2720 tablet_bootstrap.cc:492] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: No bootstrap required, opened a new log
I20260925 13:12:24.944782 2720 ts_tablet_manager.cc:1397] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:24.944880 2722 tablet_bootstrap.cc:492] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: No bootstrap required, opened a new log
I20260925 13:12:24.944934 2722 ts_tablet_manager.cc:1397] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:24.944918 2720 raft_consensus.cc:348] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.945008 2721 tablet_bootstrap.cc:654] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:24.945055 2720 raft_consensus.cc:374] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:24.945047 2722 raft_consensus.cc:348] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.945099 2720 raft_consensus.cc:729] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Initialized, Role: FOLLOWER
I20260925 13:12:24.945111 2722 raft_consensus.cc:374] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:24.945132 2722 raft_consensus.cc:729] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Initialized, Role: FOLLOWER
I20260925 13:12:24.945187 2720 consensus_queue.cc:260] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.945222 2722 consensus_queue.cc:260] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.945369 2722 ts_tablet_manager.cc:1428] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:24.945421 2720 ts_tablet_manager.cc:1428] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:24.945833 2721 tablet_bootstrap.cc:492] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: No bootstrap required, opened a new log
I20260925 13:12:24.945940 2721 ts_tablet_manager.cc:1397] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:24.946142 2721 raft_consensus.cc:348] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.946277 2721 raft_consensus.cc:374] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:24.946322 2721 raft_consensus.cc:729] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257466f3e6134234bd1f0c947200cffa, State: Initialized, Role: FOLLOWER
I20260925 13:12:24.946403 2721 consensus_queue.cc:260] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.946607 2721 ts_tablet_manager.cc:1428] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:24.992626 2695 raft_consensus.cc:482] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:24.992717 2695 raft_consensus.cc:504] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.992887 2695 leader_election.cc:288] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763)
I20260925 13:12:24.993052 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab27bf1cb547e493a11a89f7aff22f" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db77501ecdc146ef8bb5f256293571c3" is_pre_election: true
I20260925 13:12:24.993073 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab27bf1cb547e493a11a89f7aff22f" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa" is_pre_election: true
I20260925 13:12:24.993158 2549 raft_consensus.cc:2454] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 0.
I20260925 13:12:24.993158 2284 raft_consensus.cc:2454] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 0.
I20260925 13:12:24.993383 2356 leader_election.cc:302] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d9a8c5c6a045c887d2f26a7d24ce25, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:24.993502 2695 raft_consensus.cc:2781] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:24.993541 2695 raft_consensus.cc:482] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:24.993561 2695 raft_consensus.cc:3037] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:24.994321 2695 raft_consensus.cc:504] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.994468 2695 leader_election.cc:288] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 election: Requested vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763)
I20260925 13:12:24.994611 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab27bf1cb547e493a11a89f7aff22f" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa"
I20260925 13:12:24.994649 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab27bf1cb547e493a11a89f7aff22f" candidate_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db77501ecdc146ef8bb5f256293571c3"
I20260925 13:12:24.994697 2284 raft_consensus.cc:3037] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:24.994719 2549 raft_consensus.cc:3037] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:24.995414 2549 raft_consensus.cc:2454] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 1.
I20260925 13:12:24.995412 2284 raft_consensus.cc:2454] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d9a8c5c6a045c887d2f26a7d24ce25 in term 1.
I20260925 13:12:24.995590 2356 leader_election.cc:302] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d9a8c5c6a045c887d2f26a7d24ce25, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:24.995683 2695 raft_consensus.cc:2781] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:24.995740 2695 raft_consensus.cc:686] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 LEADER]: Becoming Leader. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Running, Role: LEADER
I20260925 13:12:24.995798 2695 consensus_queue.cc:237] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:24.996426 2127 catalog_manager.cc:5495] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:25.006759 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:25.004362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.98.2:34441)
0925 13:12:25.004493 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:25.004495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:25.004502 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:25.004522 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:25.004523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:25.004534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:25.004616 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:25.004755 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.005241 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.006117 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.006329 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.006404 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:25.006546 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:25.006577 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:25.006626 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:25.006680 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 13:12:25.006702 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260925 13:12:25.011518 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:25.008721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.2:34919)
0925 13:12:25.008863 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:25.008872 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:25.008940 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:25.009045 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:25.009046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:25.009047 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:25.009086 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:25.009087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.009784 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.009785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:25.010733 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:25.010736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.011080 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.011081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:25.011181 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:25.011185 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:25.011364 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:25.011365 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:12:25.011389 (+ 24us) client_negotiation.cc:240] Negotiation successful
0925 13:12:25.011424 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260925 13:12:25.011528 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:25.008776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34919 (local address 127.0.98.2:34441)
0925 13:12:25.008826 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:25.008828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:25.008925 (+ 97us) server_negotiation.cc:407] Connection header received
0925 13:12:25.008993 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:25.008994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:25.009003 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:25.009043 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:25.009130 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.009656 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.010819 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.010994 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.011050 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:25.011264 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:25.011298 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:25.011366 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:25.011393 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:12:25.011418 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 13:12:25.013306 2284 raft_consensus.cc:1261] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Refusing update from remote peer 45d9a8c5c6a045c887d2f26a7d24ce25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:25.013309 2549 raft_consensus.cc:1261] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Refusing update from remote peer 45d9a8c5c6a045c887d2f26a7d24ce25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:25.013558 2687 consensus_queue.cc:1035] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:25.013648 2690 consensus_queue.cc:1035] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:25.024017 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:25.024099 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:25.024701 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Sending DeleteTablet for 3 replicas of tablet 5bab27bf1cb547e493a11a89f7aff22f
I20260925 13:12:25.025003 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bab27bf1cb547e493a11a89f7aff22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55844
I20260925 13:12:25.025003 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bab27bf1cb547e493a11a89f7aff22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44626
I20260925 13:12:25.025094 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bab27bf1cb547e493a11a89f7aff22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52908
I20260925 13:12:25.025172 2709 tablet_replica.cc:330] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: stopping tablet replica
I20260925 13:12:25.025172 2710 tablet_replica.cc:330] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: stopping tablet replica
I20260925 13:12:25.025223 2709 raft_consensus.cc:2229] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:25.025226 2710 raft_consensus.cc:2229] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:25.025156 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 13:12:25.025260 2710 raft_consensus.cc:2258] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 13:12:25.025288 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:25.025305 2709 raft_consensus.cc:2258] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:25.025375 2711 tablet_replica.cc:330] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: stopping tablet replica
I20260925 13:12:25.025430 2711 raft_consensus.cc:2229] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:25.025465 2711 raft_consensus.cc:2258] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:25.025868 2710 ts_tablet_manager.cc:1905] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:25.026523 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 9458951fcd1b48ac8d05a2d91938a2af (DEFAULT_TABLE table=test-table [id=84588d49143b48b98c71d1d2672962d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:25.026631 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9458951fcd1b48ac8d05a2d91938a2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:25.027591 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 9458951fcd1b48ac8d05a2d91938a2af (DEFAULT_TABLE table=test-table [id=84588d49143b48b98c71d1d2672962d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:25.027696 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 9458951fcd1b48ac8d05a2d91938a2af (DEFAULT_TABLE table=test-table [id=84588d49143b48b98c71d1d2672962d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:25.027781 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9458951fcd1b48ac8d05a2d91938a2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:25.027980 2721 tablet_bootstrap.cc:492] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa: Bootstrap starting.
I20260925 13:12:25.028438 2721 tablet_bootstrap.cc:654] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:25.028889 2722 tablet_bootstrap.cc:492] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Bootstrap starting.
I20260925 13:12:25.029124 2709 ts_tablet_manager.cc:1905] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:25.029181 2722 tablet_bootstrap.cc:654] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:25.030118 2722 tablet_bootstrap.cc:492] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: No bootstrap required, opened a new log
I20260925 13:12:25.030160 2722 ts_tablet_manager.cc:1397] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:25.030278 2722 raft_consensus.cc:348] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.030349 2722 raft_consensus.cc:374] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:25.030376 2722 raft_consensus.cc:729] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d9a8c5c6a045c887d2f26a7d24ce25, State: Initialized, Role: FOLLOWER
I20260925 13:12:25.030421 2722 consensus_queue.cc:260] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.030519 2722 ts_tablet_manager.cc:1428] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:25.030608 2709 ts_tablet_manager.cc:1918] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:25.030654 2709 log.cc:1196] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/5bab27bf1cb547e493a11a89f7aff22f
I20260925 13:12:25.030805 2709 ts_tablet_manager.cc:1939] T 5bab27bf1cb547e493a11a89f7aff22f P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting consensus metadata
I20260925 13:12:25.031195 2710 ts_tablet_manager.cc:1918] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:25.031242 2710 log.cc:1196] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/5bab27bf1cb547e493a11a89f7aff22f
I20260925 13:12:25.031414 2710 ts_tablet_manager.cc:1939] T 5bab27bf1cb547e493a11a89f7aff22f P db77501ecdc146ef8bb5f256293571c3: Deleting consensus metadata
I20260925 13:12:25.031553 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9458951fcd1b48ac8d05a2d91938a2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:25.032362 2720 tablet_bootstrap.cc:492] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Bootstrap starting.
I20260925 13:12:25.032441 2112 catalog_manager.cc:4841] TS db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763): tablet 5bab27bf1cb547e493a11a89f7aff22f (table test-table [id=37d91a2833f04ed394fa4754e0388fd9]) successfully deleted
I20260925 13:12:25.032732 2720 tablet_bootstrap.cc:654] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:25.032809 2111 catalog_manager.cc:4841] TS 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441): tablet 5bab27bf1cb547e493a11a89f7aff22f (table test-table [id=37d91a2833f04ed394fa4754e0388fd9]) successfully deleted
I20260925 13:12:25.032941 2711 ts_tablet_manager.cc:1905] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:25.033293 2720 tablet_bootstrap.cc:492] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: No bootstrap required, opened a new log
I20260925 13:12:25.033341 2720 ts_tablet_manager.cc:1397] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:25.033478 2720 raft_consensus.cc:348] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.033535 2720 raft_consensus.cc:374] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:25.033555 2720 raft_consensus.cc:729] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Initialized, Role: FOLLOWER
I20260925 13:12:25.033597 2720 consensus_queue.cc:260] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.033842 2720 ts_tablet_manager.cc:1428] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:25.034343 2711 ts_tablet_manager.cc:1918] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:25.034403 2711 log.cc:1196] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/5bab27bf1cb547e493a11a89f7aff22f
I20260925 13:12:25.034647 2711 ts_tablet_manager.cc:1939] T 5bab27bf1cb547e493a11a89f7aff22f P 257466f3e6134234bd1f0c947200cffa: Deleting consensus metadata
I20260925 13:12:25.034972 2111 catalog_manager.cc:4841] TS 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455): tablet 5bab27bf1cb547e493a11a89f7aff22f (table test-table [id=37d91a2833f04ed394fa4754e0388fd9]) successfully deleted
I20260925 13:12:25.035159 2721 tablet_bootstrap.cc:492] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa: No bootstrap required, opened a new log
I20260925 13:12:25.035212 2721 ts_tablet_manager.cc:1397] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 13:12:25.035341 2721 raft_consensus.cc:348] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.035416 2721 raft_consensus.cc:374] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:25.035437 2721 raft_consensus.cc:729] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257466f3e6134234bd1f0c947200cffa, State: Initialized, Role: FOLLOWER
I20260925 13:12:25.035486 2721 consensus_queue.cc:260] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.035575 2721 ts_tablet_manager.cc:1428] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:25.043872 2703 raft_consensus.cc:482] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:25.043949 2703 raft_consensus.cc:504] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.044075 2703 leader_election.cc:288] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441)
I20260925 13:12:25.044284 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9458951fcd1b48ac8d05a2d91938a2af" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa" is_pre_election: true
I20260925 13:12:25.044276 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9458951fcd1b48ac8d05a2d91938a2af" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" is_pre_election: true
I20260925 13:12:25.044363 2424 raft_consensus.cc:2454] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 0.
I20260925 13:12:25.044363 2284 raft_consensus.cc:2454] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 0.
I20260925 13:12:25.044502 2495 leader_election.cc:302] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257466f3e6134234bd1f0c947200cffa, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:25.044616 2703 raft_consensus.cc:2781] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:25.044651 2703 raft_consensus.cc:482] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:25.044670 2703 raft_consensus.cc:3037] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:25.045445 2703 raft_consensus.cc:504] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.045560 2703 leader_election.cc:288] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 election: Requested vote from peers 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455), 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441)
I20260925 13:12:25.045768 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9458951fcd1b48ac8d05a2d91938a2af" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25"
I20260925 13:12:25.045768 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9458951fcd1b48ac8d05a2d91938a2af" candidate_uuid: "db77501ecdc146ef8bb5f256293571c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257466f3e6134234bd1f0c947200cffa"
I20260925 13:12:25.045827 2284 raft_consensus.cc:3037] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:25.045827 2424 raft_consensus.cc:3037] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:25.046525 2424 raft_consensus.cc:2454] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 1.
I20260925 13:12:25.046525 2284 raft_consensus.cc:2454] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77501ecdc146ef8bb5f256293571c3 in term 1.
I20260925 13:12:25.046681 2495 leader_election.cc:302] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257466f3e6134234bd1f0c947200cffa, db77501ecdc146ef8bb5f256293571c3; no voters:
I20260925 13:12:25.046783 2703 raft_consensus.cc:2781] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:25.046830 2703 raft_consensus.cc:686] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 1 LEADER]: Becoming Leader. State: Replica: db77501ecdc146ef8bb5f256293571c3, State: Running, Role: LEADER
I20260925 13:12:25.046875 2703 consensus_queue.cc:237] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } }
I20260925 13:12:25.047437 2127 catalog_manager.cc:5495] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db77501ecdc146ef8bb5f256293571c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "db77501ecdc146ef8bb5f256293571c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db77501ecdc146ef8bb5f256293571c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35763 } health_report { overall_health: HEALTHY } } }
I20260925 13:12:25.059720 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:25.057130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.0.98.3:35763)
0925 13:12:25.057293 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:25.057295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:25.057303 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:25.057326 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:25.057330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:25.057341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:25.057396 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:25.057537 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.058100 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.059166 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:25.059356 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:25.059405 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:25.059545 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:25.059569 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:25.059601 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:25.059649 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 13:12:25.059663 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":119,"threads_started":1}
I20260925 13:12:25.062116 2424 raft_consensus.cc:1261] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Refusing update from remote peer db77501ecdc146ef8bb5f256293571c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:25.062119 2284 raft_consensus.cc:1261] T 9458951fcd1b48ac8d05a2d91938a2af P 257466f3e6134234bd1f0c947200cffa [term 1 FOLLOWER]: Refusing update from remote peer db77501ecdc146ef8bb5f256293571c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:25.062330 2640 consensus_queue.cc:1035] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d9a8c5c6a045c887d2f26a7d24ce25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:25.062448 2703 consensus_queue.cc:1035] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257466f3e6134234bd1f0c947200cffa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:12:25.066529 2692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 13:12:26.584414 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:26.584507 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:26.585258 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef21f788b6554074aa6101e69498c062: Sending DeleteTablet for 3 replicas of tablet 9458951fcd1b48ac8d05a2d91938a2af
I20260925 13:12:26.585554 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 9458951fcd1b48ac8d05a2d91938a2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55844
I20260925 13:12:26.585613 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 9458951fcd1b48ac8d05a2d91938a2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44626
I20260925 13:12:26.585616 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 9458951fcd1b48ac8d05a2d91938a2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52908
I20260925 13:12:26.585757 2766 tablet_replica.cc:330] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: stopping tablet replica
I20260925 13:12:26.585811 2768 tablet_replica.cc:330] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: stopping tablet replica
I20260925 13:12:26.585816 2766 raft_consensus.cc:2229] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:26.585853 2768 raft_consensus.cc:2229] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:26.585866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 2199
I20260925 13:12:26.585906 2768 raft_consensus.cc:2258] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:26.585850 2766 raft_consensus.cc:2258] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:26.586736 2768 ts_tablet_manager.cc:1905] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:26.587641 2766 ts_tablet_manager.cc:1905] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:26.588209 2768 ts_tablet_manager.cc:1918] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:12:26.588256 2768 log.cc:1196] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/9458951fcd1b48ac8d05a2d91938a2af
I20260925 13:12:26.588511 2768 ts_tablet_manager.cc:1939] T 9458951fcd1b48ac8d05a2d91938a2af P db77501ecdc146ef8bb5f256293571c3: Deleting consensus metadata
I20260925 13:12:26.588804 2112 catalog_manager.cc:4841] TS db77501ecdc146ef8bb5f256293571c3 (127.0.98.3:35763): tablet 9458951fcd1b48ac8d05a2d91938a2af (table test-table [id=84588d49143b48b98c71d1d2672962d3]) successfully deleted
I20260925 13:12:26.589206 2766 ts_tablet_manager.cc:1918] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:12:26.589254 2766 log.cc:1196] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/9458951fcd1b48ac8d05a2d91938a2af
I20260925 13:12:26.589459 2766 ts_tablet_manager.cc:1939] T 9458951fcd1b48ac8d05a2d91938a2af P 45d9a8c5c6a045c887d2f26a7d24ce25: Deleting consensus metadata
I20260925 13:12:26.589787 2111 catalog_manager.cc:4841] TS 45d9a8c5c6a045c887d2f26a7d24ce25 (127.0.98.2:34441): tablet 9458951fcd1b48ac8d05a2d91938a2af (table test-table [id=84588d49143b48b98c71d1d2672962d3]) successfully deleted
W20260925 13:12:26.592554 2111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45455 (error 108)
I20260925 13:12:26.592792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 2339
I20260925 13:12:26.594642 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:26.592901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45455 (local address 127.0.0.1:40398)
0925 13:12:26.594541 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:26.594594 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1613,"thread_start_us":81,"threads_started":1}
W20260925 13:12:26.594812 2111 catalog_manager.cc:4584] TS 257466f3e6134234bd1f0c947200cffa (127.0.98.1:45455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9458951fcd1b48ac8d05a2d91938a2af: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45455: connect: Connection refused (error 111)
I20260925 13:12:26.599188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 2479
I20260925 13:12:26.604301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 2095
2026-09-25T13:12:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5469 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:26 dist-test-slave-2d0s krb5kdc[2773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:26 dist-test-slave-2d0s krb5kdc[2773](info): set up 2 sockets
Sep 25 13:12:26 dist-test-slave-2d0s krb5kdc[2773](info): commencing operation
krb5kdc: starting...
W20260925 13:12:27.621348 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:27 dist-test-slave-2d0s krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:27.732210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44597
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:27.809331 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:27.809540 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:27.809576 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:27.810683 2786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:27.810748 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:27.810771 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:27.810792 2786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:27.810822 2786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:27.812260 2786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:27.812517 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:27.812722 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:27.817203 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:27.817247 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:27.817381 2786 server_base.cc:1034] running on GCE node
W20260925 13:12:27.817205 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:27.817700 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:27.817997 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:27.819170 2786 hybrid_clock.cc:648] HybridClock initialized: now 1790341947819161 us; error 39 us; skew 500 ppm
Sep 25 13:12:27 dist-test-slave-2d0s krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:27.821542 2786 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:27.822062 2786 webserver.cc:460] Webserver started at http://127.0.98.62:38667/ using document root <none> and password file <none>
I20260925 13:12:27.822294 2786 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:27.822371 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:27.822518 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:27.823613 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "06d6e1728a894c3ca0404dae8ede0f37"
format_stamp: "Formatted at 2026-09-25 13:12:27 on dist-test-slave-2d0s"
server_key: "cca311caea647f0eabc0557760c76873"
server_key_iv: "1b7a25ea3dcce73ae8aeabae4e438c5b"
server_key_version: "encryptionkey@0"
I20260925 13:12:27.824023 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "06d6e1728a894c3ca0404dae8ede0f37"
format_stamp: "Formatted at 2026-09-25 13:12:27 on dist-test-slave-2d0s"
server_key: "cca311caea647f0eabc0557760c76873"
server_key_iv: "1b7a25ea3dcce73ae8aeabae4e438c5b"
server_key_version: "encryptionkey@0"
I20260925 13:12:27.825500 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:27.826429 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:27.826627 2786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:27.826742 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "06d6e1728a894c3ca0404dae8ede0f37"
format_stamp: "Formatted at 2026-09-25 13:12:27 on dist-test-slave-2d0s"
server_key: "cca311caea647f0eabc0557760c76873"
server_key_iv: "1b7a25ea3dcce73ae8aeabae4e438c5b"
server_key_version: "encryptionkey@0"
I20260925 13:12:27.826822 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:27.849030 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:27.850934 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:27.851111 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:27.855468 2786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44597
I20260925 13:12:27.855509 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44597 every 8 connection(s)
I20260925 13:12:27.856051 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:27.856166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 2786
I20260925 13:12:27.856261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:27.856494 392 external_mini_cluster.cc:1428] Setting key e6893be0c04e552481ea7f5d4aed4259
I20260925 13:12:27.856877 2854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:27.859979 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37: Bootstrap starting.
I20260925 13:12:27.860635 2854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:27.860924 2854 log.cc:824] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:27.861627 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37: No bootstrap required, opened a new log
Sep 25 13:12:27 dist-test-slave-2d0s krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:27.864144 2854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6e1728a894c3ca0404dae8ede0f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44597 } }
I20260925 13:12:27.864300 2854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:27.864343 2854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d6e1728a894c3ca0404dae8ede0f37, State: Initialized, Role: FOLLOWER
I20260925 13:12:27.864687 2854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6e1728a894c3ca0404dae8ede0f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44597 } }
I20260925 13:12:27.864789 2854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:27.864831 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:27.864924 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:27.865432 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6e1728a894c3ca0404dae8ede0f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44597 } }
I20260925 13:12:27.865512 2854 leader_election.cc:302] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d6e1728a894c3ca0404dae8ede0f37; no voters:
I20260925 13:12:27.865693 2854 leader_election.cc:288] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:27.865723 2859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:27.865839 2859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [term 1 LEADER]: Becoming Leader. State: Replica: 06d6e1728a894c3ca0404dae8ede0f37, State: Running, Role: LEADER
I20260925 13:12:27.866017 2859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6e1728a894c3ca0404dae8ede0f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44597 } }
I20260925 13:12:27.866000 2854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 13:12:27.868196 2874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 13:12:27.868335 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d6e1728a894c3ca0404dae8ede0f37. Latest consensus state: current_term: 1 leader_uuid: "06d6e1728a894c3ca0404dae8ede0f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6e1728a894c3ca0404dae8ede0f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44597 } } }
W20260925 13:12:27.868413 2874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:27.868587 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:27.857030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.0.98.62:44597)
0925 13:12:27.857230 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:27.857239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:27.857277 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:12:27.858023 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:27.858030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:27.858081 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:27.858203 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:27.858975 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:27.859905 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:27.860939 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:27.861403 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:27.861492 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:27.864117 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:27.864121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:27.864121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:27.864134 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:27.866141 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:27.866666 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:27.866669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:27.866670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:27.866778 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:27.867083 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:27.867084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:27.867084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:27.867426 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:27.868130 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:27.868209 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 13:12:27.868346 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260925 13:12:27.868394 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d6e1728a894c3ca0404dae8ede0f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6e1728a894c3ca0404dae8ede0f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44597 } } }
I20260925 13:12:27.868769 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:27.868505 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:27.869122 2876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:27.869370 2876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:27.870862 2876 catalog_manager.cc:1347] Generated new cluster ID: e56202b04efd4003a504f973055e8173
I20260925 13:12:27.870913 2876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:27.877000 2876 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:27.877470 2876 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:27.879796 2876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d6e1728a894c3ca0404dae8ede0f37: Generated new TSK 0
I20260925 13:12:27.879966 2876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:27.913072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44597
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:28.039551 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:28.039835 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:28.039875 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:28.041616 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:28.041718 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:28.041766 2880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:28.044062 2880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:28.044313 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:28.044572 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:28.049911 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:28.050004 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:28.050107 2880 server_base.cc:1034] running on GCE node
W20260925 13:12:28.049898 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:28.050369 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:28.050618 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:28.051777 2880 hybrid_clock.cc:648] HybridClock initialized: now 1790341948051766 us; error 35 us; skew 500 ppm
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:28.057842 2880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:28.058286 2880 webserver.cc:460] Webserver started at http://127.0.98.1:43965/ using document root <none> and password file <none>
I20260925 13:12:28.058504 2880 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:28.058568 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:28.058698 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:28.059664 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "1900ea06ce7d4358bddeb1c1405cba0c"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "9f46228d38b280562c8df0d8b2bd00a4"
server_key_iv: "b011093fb0d8f320d146ba93f222f35a"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.060017 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1900ea06ce7d4358bddeb1c1405cba0c"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "9f46228d38b280562c8df0d8b2bd00a4"
server_key_iv: "b011093fb0d8f320d146ba93f222f35a"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.061349 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:28.062160 2895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.062325 2880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:28.062439 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "1900ea06ce7d4358bddeb1c1405cba0c"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "9f46228d38b280562c8df0d8b2bd00a4"
server_key_iv: "b011093fb0d8f320d146ba93f222f35a"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.062510 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:28.080392 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:28.082851 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:28.083014 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:28.083245 2880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:28.083540 2880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:28.083595 2880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.083633 2880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:28.083660 2880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.089493 2880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36079
I20260925 13:12:28.089543 3008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36079 every 8 connection(s)
I20260925 13:12:28.089875 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:28.098383 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:28.090583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44165 (local address 127.0.98.62:44597)
0925 13:12:28.090626 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:28.090628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:28.090821 (+ 193us) server_negotiation.cc:407] Connection header received
0925 13:12:28.091719 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:28.091720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:28.091729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:28.091784 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:28.092405 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.092906 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.093878 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.094072 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.094125 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:28.096063 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:28.096065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:28.096065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:28.096071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:28.097232 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:28.097573 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:28.097574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:28.097574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:28.097622 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:28.097875 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:28.097875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:28.097876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:28.097973 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:28.098024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:28.098240 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 13:12:28.098306 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:28.098572 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:28.090542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44597 (local address 127.0.98.1:44165)
0925 13:12:28.090695 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:28.090735 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:28.091630 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:28.091878 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:28.091882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:28.092137 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:28.092313 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:28.092318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.092985 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.092985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:28.093805 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:28.093807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.094140 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.094141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:28.094229 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:28.094703 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:28.094710 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:28.095977 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:28.097299 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:28.097304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:28.097306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:28.097508 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:28.097715 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:28.097716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:28.097716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:28.097819 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:28.098033 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:28.098037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:28.098137 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 13:12:28.098214 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 13:12:28.098319 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260925 13:12:28.098558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 2880
I20260925 13:12:28.098651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:28.098824 392 external_mini_cluster.cc:1428] Setting key b56c08a71298aa7c06a7daf298972a8e
I20260925 13:12:28.099104 3009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44597
I20260925 13:12:28.099189 3009 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:28.099363 3009 heartbeater.cc:511] Master 127.0.98.62:44597 requested a full tablet report, sending...
I20260925 13:12:28.099859 2818 ts_manager.cc:194] Registered new tserver with Master: 1900ea06ce7d4358bddeb1c1405cba0c (127.0.98.1:36079)
I20260925 13:12:28.100428 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44165
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:28.122351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44597
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:28.200045 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:28.200222 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:28.200241 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:28.201304 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:28.201344 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:28.201368 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:28.202812 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:28.203049 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:28.203266 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:12:28.208218 3014 server_base.cc:1034] running on GCE node
W20260925 13:12:28.208181 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:28.208254 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:28.208189 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:28.208633 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:28.208825 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:28.209959 3014 hybrid_clock.cc:648] HybridClock initialized: now 1790341948209941 us; error 41 us; skew 500 ppm
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:28.212419 3014 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:28.212903 3014 webserver.cc:460] Webserver started at http://127.0.98.2:42161/ using document root <none> and password file <none>
I20260925 13:12:28.213135 3014 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:28.213202 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:28.213347 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:28.214432 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "2369f4ab4a61431aaaa9676ff89aa6b4"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "910c6bc83154485989fa2b8ea30a271c"
server_key_iv: "5d4920f02e06ec4f851d93545d45d344"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.214807 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2369f4ab4a61431aaaa9676ff89aa6b4"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "910c6bc83154485989fa2b8ea30a271c"
server_key_iv: "5d4920f02e06ec4f851d93545d45d344"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.216090 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:28.216853 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.217025 3014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:28.217139 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "2369f4ab4a61431aaaa9676ff89aa6b4"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "910c6bc83154485989fa2b8ea30a271c"
server_key_iv: "5d4920f02e06ec4f851d93545d45d344"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.217211 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:28.235172 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:28.237494 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:28.237656 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:28.237946 3014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:28.238270 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:28.238324 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.238365 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:28.238392 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.244342 3014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33723
I20260925 13:12:28.244395 3142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33723 every 8 connection(s)
I20260925 13:12:28.244697 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:12:28.246610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3014
I20260925 13:12:28.246713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:28.246907 392 external_mini_cluster.cc:1428] Setting key bb2641e21b7e6273a3d001a489200d36
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:12:28.255055 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:28.245346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35255 (local address 127.0.98.62:44597)
0925 13:12:28.245393 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:28.245395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:28.245622 (+ 227us) server_negotiation.cc:407] Connection header received
0925 13:12:28.246718 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:28.246720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:28.246732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:28.246798 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:28.247573 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.248323 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.249305 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.249630 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.249731 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:28.252106 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:28.252107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:28.252108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:28.252115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:28.253542 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:28.254005 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:28.254006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:28.254006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:28.254084 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:28.254354 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:28.254354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:28.254355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:28.254469 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:28.254538 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:28.254863 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 13:12:28.254958 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:28.255213 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:28.245342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44597 (local address 127.0.98.2:35255)
0925 13:12:28.245499 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:28.245533 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:28.246584 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:28.246884 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:28.246889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:28.247193 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:28.247469 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:28.247476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.248423 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.248424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:28.249219 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:28.249221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.249746 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.249747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:28.249857 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:28.250356 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:28.250364 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:28.252011 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:28.253646 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:28.253653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:28.253655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:28.253908 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:28.254152 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:28.254153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:28.254153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:28.254278 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:28.254541 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:28.254548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:28.254697 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 13:12:28.254826 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 13:12:28.254997 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260925 13:12:28.255677 3143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44597
I20260925 13:12:28.255756 3143 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:28.255949 3143 heartbeater.cc:511] Master 127.0.98.62:44597 requested a full tablet report, sending...
I20260925 13:12:28.256358 2818 ts_manager.cc:194] Registered new tserver with Master: 2369f4ab4a61431aaaa9676ff89aa6b4 (127.0.98.2:33723)
I20260925 13:12:28.256713 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:28.270402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44597
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:28.349251 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:28.349478 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:28.349515 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:28.350646 3148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:28.350710 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:28.350751 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:28.352259 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:28.352511 3148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:28.352722 3148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:28.357707 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:28.357723 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:28.357721 3156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:28.357918 3148 server_base.cc:1034] running on GCE node
I20260925 13:12:28.358125 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:28.358346 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:28.359483 3148 hybrid_clock.cc:648] HybridClock initialized: now 1790341948359469 us; error 29 us; skew 500 ppm
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:28.361809 3148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:28.362179 3148 webserver.cc:460] Webserver started at http://127.0.98.3:34825/ using document root <none> and password file <none>
I20260925 13:12:28.362380 3148 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:28.362419 3148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:28.362547 3148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:28.363538 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "75587e0f5e35434d9a77c8db1418f326"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "b1f8586336072d87e749d481e1e4d718"
server_key_iv: "dbbba7b5827a9cac7c0ece4dd0309030"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.363883 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75587e0f5e35434d9a77c8db1418f326"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "b1f8586336072d87e749d481e1e4d718"
server_key_iv: "dbbba7b5827a9cac7c0ece4dd0309030"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.365236 3148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:28.366003 3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.366170 3148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.366294 3148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "75587e0f5e35434d9a77c8db1418f326"
format_stamp: "Formatted at 2026-09-25 13:12:28 on dist-test-slave-2d0s"
server_key: "b1f8586336072d87e749d481e1e4d718"
server_key_iv: "dbbba7b5827a9cac7c0ece4dd0309030"
server_key_version: "encryptionkey@0"
I20260925 13:12:28.366380 3148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:28.376936 3148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:28.378976 3148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:28.379156 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:28.379385 3148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:28.379691 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:28.379747 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.379794 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:28.379840 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:28.386071 3148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44657
I20260925 13:12:28.386104 3276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44657 every 8 connection(s)
I20260925 13:12:28.386461 3148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:28.394837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3148
I20260925 13:12:28.394950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:28.395156 392 external_mini_cluster.cc:1428] Setting key 9bd272491c2d07adcd63feabcbcefd32
I20260925 13:12:28.396198 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:28.387209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35933 (local address 127.0.98.62:44597)
0925 13:12:28.387286 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:28.387288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:28.387505 (+ 217us) server_negotiation.cc:407] Connection header received
0925 13:12:28.388480 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:28.388481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:28.388508 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:28.388562 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:28.389187 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.389910 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.390885 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.391072 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.391121 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:28.393293 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:28.393294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:28.393295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:28.393301 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:28.394731 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:28.395191 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:28.395192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:28.395192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:28.395273 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:28.395534 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:28.395535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:28.395536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:28.395671 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:28.395734 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:28.396052 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 13:12:28.396137 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 13:12:28.396471 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:28.387171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44597 (local address 127.0.98.3:35933)
0925 13:12:28.387350 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:28.387417 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:28.388344 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:28.388640 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:28.388644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:28.388902 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:28.389089 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:28.389093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.390016 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.390018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:28.390814 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:28.390816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:28.391150 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:28.391150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:28.391261 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:28.391705 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:28.391714 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:28.393202 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:28.394808 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:28.394815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:28.394817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:28.395117 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:28.395322 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:28.395322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:28.395323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:28.395465 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:28.395740 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:28.395745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:28.395901 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 13:12:28.396019 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 13:12:28.396172 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260925 13:12:28.396878 3277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44597
I20260925 13:12:28.396955 3277 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:28.397145 3277 heartbeater.cc:511] Master 127.0.98.62:44597 requested a full tablet report, sending...
I20260925 13:12:28.397511 2818 ts_manager.cc:194] Registered new tserver with Master: 75587e0f5e35434d9a77c8db1418f326 (127.0.98.3:44657)
I20260925 13:12:28.397949 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35933
I20260925 13:12:28.406783 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 13:12:28.417168 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:28.416287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.98.62:44597)
0925 13:12:28.416339 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:28.416341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:28.416481 (+ 140us) server_negotiation.cc:407] Connection header received
0925 13:12:28.416525 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:28.416526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:28.416535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:28.416589 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:28.417075 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43194: BlockingRecv error: recv got EOF from 127.0.0.1:43194 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:12:28.418033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 2880
I20260925 13:12:28.422307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3014
I20260925 13:12:28.426529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3148
I20260925 13:12:28.430678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 2786
2026-09-25T13:12:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1825 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[3289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[3289](info): set up 2 sockets
Sep 25 13:12:28 dist-test-slave-2d0s krb5kdc[3289](info): commencing operation
krb5kdc: starting...
W20260925 13:12:29.529745 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:29 dist-test-slave-2d0s krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:29.648372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:29.728612 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:29.728819 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:29.728840 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:29.730085 3302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:29.730127 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:29.730142 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:29.730155 3302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:29.730167 3302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:29.731657 3302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:29.731904 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:29.732152 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:29.736965 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:29.736959 3310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:29.736941 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:29.737358 3302 server_base.cc:1034] running on GCE node
I20260925 13:12:29.737627 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:29.737933 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:29.739080 3302 hybrid_clock.cc:648] HybridClock initialized: now 1790341949739070 us; error 32 us; skew 500 ppm
Sep 25 13:12:29 dist-test-slave-2d0s krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:29.741638 3302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:29.742151 3302 webserver.cc:460] Webserver started at http://127.0.98.62:40539/ using document root <none> and password file <none>
I20260925 13:12:29.742408 3302 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:29.742453 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:29.742606 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:29.743755 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "5d2b529ccb9840b78fb9b07f0a6dc605"
format_stamp: "Formatted at 2026-09-25 13:12:29 on dist-test-slave-2d0s"
server_key: "6c4c49e75b3077b734c3805f6eeb20fd"
server_key_iv: "f33d26649871c7e5ce28dbb30c614d2f"
server_key_version: "encryptionkey@0"
I20260925 13:12:29.744115 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d2b529ccb9840b78fb9b07f0a6dc605"
format_stamp: "Formatted at 2026-09-25 13:12:29 on dist-test-slave-2d0s"
server_key: "6c4c49e75b3077b734c3805f6eeb20fd"
server_key_iv: "f33d26649871c7e5ce28dbb30c614d2f"
server_key_version: "encryptionkey@0"
I20260925 13:12:29.745463 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:29.746253 3317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:29.746429 3302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:29.746531 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "5d2b529ccb9840b78fb9b07f0a6dc605"
format_stamp: "Formatted at 2026-09-25 13:12:29 on dist-test-slave-2d0s"
server_key: "6c4c49e75b3077b734c3805f6eeb20fd"
server_key_iv: "f33d26649871c7e5ce28dbb30c614d2f"
server_key_version: "encryptionkey@0"
I20260925 13:12:29.746623 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:29.755072 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:29.756819 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:29.757009 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:29.760821 3302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42977
I20260925 13:12:29.760885 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42977 every 8 connection(s)
I20260925 13:12:29.761376 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:29.762089 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:29.763190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3302
I20260925 13:12:29.763307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:29.763469 392 external_mini_cluster.cc:1428] Setting key 466663cd711a5d9d1ee9aa7544c10ad7
I20260925 13:12:29.764947 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605: Bootstrap starting.
I20260925 13:12:29.765565 3370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:29.765913 3370 log.cc:824] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:29.766667 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605: No bootstrap required, opened a new log
I20260925 13:12:29.768919 3370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42977 } }
I20260925 13:12:29.769099 3370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:29.769138 3370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2b529ccb9840b78fb9b07f0a6dc605, State: Initialized, Role: FOLLOWER
I20260925 13:12:29.769274 3370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42977 } }
I20260925 13:12:29.769335 3370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:29.769356 3370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:29.769428 3370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:29.770025 3370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42977 } }
I20260925 13:12:29.770112 3370 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d2b529ccb9840b78fb9b07f0a6dc605; no voters:
I20260925 13:12:29.770291 3370 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:29.770314 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:29.770493 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2b529ccb9840b78fb9b07f0a6dc605, State: Running, Role: LEADER
I20260925 13:12:29.770574 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42977 } }
I20260925 13:12:29.770619 3370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:29.770887 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42977 } } }
Sep 25 13:12:29 dist-test-slave-2d0s krb5kdc[3289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:29.771034 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:29.771196 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2b529ccb9840b78fb9b07f0a6dc605. Latest consensus state: current_term: 1 leader_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b529ccb9840b78fb9b07f0a6dc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42977 } } }
I20260925 13:12:29.771297 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:29.771461 3381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:29.772286 3381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:29.774231 3381 catalog_manager.cc:1347] Generated new cluster ID: 7c8a76cd146e492aa07ce1941237a2fe
I20260925 13:12:29.774279 3381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:29.775394 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:29.764023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.0.98.62:42977)
0925 13:12:29.764227 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:29.764237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:29.764267 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:12:29.764785 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:29.764794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:29.764853 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:29.765007 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:29.765752 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:29.767080 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:29.767872 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:29.768569 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:29.768686 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:29.771589 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:29.771593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:29.771593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:29.771606 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:29.773442 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:29.773864 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:29.773867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:29.773867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:29.773962 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:29.774208 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:29.774208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:29.774209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:29.774512 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:29.774641 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:29.775060 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 13:12:29.775183 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260925 13:12:29.780414 3381 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:29.780875 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:12:29.781104 3381 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:29.790438 3381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d2b529ccb9840b78fb9b07f0a6dc605: Generated new TSK 0
I20260925 13:12:29.790627 3381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:12:29.798465 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:29.794681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.98.62:42977)
0925 13:12:29.794731 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:29.794733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:29.794840 (+ 107us) server_negotiation.cc:407] Connection header received
0925 13:12:29.794902 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:29.794903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:29.794912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:29.794971 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:29.795411 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:29.795909 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:29.797025 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:29.797238 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:29.797287 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:29.798016 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:29.798017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:29.798018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:29.798200 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:29.798216 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:29.798221 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:29.798266 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:29.798349 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 13:12:29.798375 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:12:29.800809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3302
2026-09-25T13:12:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1368 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:29 dist-test-slave-2d0s krb5kdc[3403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:29 dist-test-slave-2d0s krb5kdc[3403](info): set up 2 sockets
Sep 25 13:12:29 dist-test-slave-2d0s krb5kdc[3403](info): commencing operation
krb5kdc: starting...
W20260925 13:12:30.817574 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:30 dist-test-slave-2d0s krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:30.934574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45759
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:31.016007 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:31.016256 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:31.016281 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:31.017690 3416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:31.017745 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:31.017760 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:31.017773 3416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:31.017786 3416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:31.019652 3416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:31.019940 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:31.020192 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:31.025368 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.025389 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:31.025418 3416 server_base.cc:1034] running on GCE node
W20260925 13:12:31.025384 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:31.025830 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:31.026111 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:31.027359 3416 hybrid_clock.cc:648] HybridClock initialized: now 1790341951027345 us; error 30 us; skew 500 ppm
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:31.030121 3416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:31.030531 3416 webserver.cc:460] Webserver started at http://127.0.98.62:43605/ using document root <none> and password file <none>
I20260925 13:12:31.030732 3416 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:31.030774 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:31.030865 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:31.031926 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "5386d44925ca4e639008902d83cf3d94"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "5444ef227b0b3b70b00cb7d3b29b89b7"
server_key_iv: "ccea2ed75366a21a92d868ed0b1f5689"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.032240 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "5386d44925ca4e639008902d83cf3d94"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "5444ef227b0b3b70b00cb7d3b29b89b7"
server_key_iv: "ccea2ed75366a21a92d868ed0b1f5689"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.033514 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:31.034302 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.034507 3416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:31.034587 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "5386d44925ca4e639008902d83cf3d94"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "5444ef227b0b3b70b00cb7d3b29b89b7"
server_key_iv: "ccea2ed75366a21a92d868ed0b1f5689"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.034639 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:31.047885 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:31.049750 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:31.049861 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:31.053442 3416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45759
I20260925 13:12:31.053543 3483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45759 every 8 connection(s)
I20260925 13:12:31.053908 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:31.054910 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:31.057181 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94: Bootstrap starting.
I20260925 13:12:31.057813 3484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:31.058151 3484 log.cc:824] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:31.058866 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94: No bootstrap required, opened a new log
I20260925 13:12:31.059315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3416
I20260925 13:12:31.059413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:31.059615 392 external_mini_cluster.cc:1428] Setting key 7e6ec5085121115a9a269df998b1a39d
I20260925 13:12:31.061020 3484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5386d44925ca4e639008902d83cf3d94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45759 } }
I20260925 13:12:31.061179 3484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:31.061241 3484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5386d44925ca4e639008902d83cf3d94, State: Initialized, Role: FOLLOWER
I20260925 13:12:31.061398 3484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5386d44925ca4e639008902d83cf3d94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45759 } }
I20260925 13:12:31.061506 3484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:31.061553 3484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:31.061604 3484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:31.062466 3484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5386d44925ca4e639008902d83cf3d94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45759 } }
I20260925 13:12:31.062584 3484 leader_election.cc:302] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5386d44925ca4e639008902d83cf3d94; no voters:
I20260925 13:12:31.062777 3484 leader_election.cc:288] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:31.062793 3489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:31.062899 3489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [term 1 LEADER]: Becoming Leader. State: Replica: 5386d44925ca4e639008902d83cf3d94, State: Running, Role: LEADER
I20260925 13:12:31.063031 3484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:31.063031 3489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5386d44925ca4e639008902d83cf3d94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45759 } }
I20260925 13:12:31.063500 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5386d44925ca4e639008902d83cf3d94. Latest consensus state: current_term: 1 leader_uuid: "5386d44925ca4e639008902d83cf3d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5386d44925ca4e639008902d83cf3d94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45759 } } }
I20260925 13:12:31.063482 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5386d44925ca4e639008902d83cf3d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5386d44925ca4e639008902d83cf3d94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45759 } } }
I20260925 13:12:31.063714 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:31.063700 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:31.063910 3498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:31.064497 3498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:31.066591 3498 catalog_manager.cc:1347] Generated new cluster ID: 7944e11e12ad4e7cb8e0645470b46019
I20260925 13:12:31.066643 3498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:31.071300 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.060193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.0.98.62:45759)
0925 13:12:31.060405 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:31.060444 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:31.060471 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:12:31.061033 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:31.061041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:31.061095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:31.061290 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:31.062032 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.063092 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.064232 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.064838 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.064956 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.067679 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:31.067683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:31.067683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:31.067695 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:31.069706 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.070079 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.070082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.070082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.070208 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.070403 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.070404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.070404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.070710 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:31.070869 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:31.070933 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 13:12:31.071064 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260925 13:12:31.078055 3498 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:31.078625 3498 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:31.081498 3498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5386d44925ca4e639008902d83cf3d94: Generated new TSK 0
I20260925 13:12:31.081647 3498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:31.108673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45759
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:31.188860 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:31.189052 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:31.189074 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:31.190282 3510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:31.190325 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:31.190351 3510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:31.192314 3510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:31.192618 3510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:31.192866 3510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:31.197649 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.197682 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.197752 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:31.198048 3510 server_base.cc:1034] running on GCE node
I20260925 13:12:31.198252 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:31.198480 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:31.199678 3510 hybrid_clock.cc:648] HybridClock initialized: now 1790341951199673 us; error 35 us; skew 500 ppm
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:31.202097 3510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:31.202579 3510 webserver.cc:460] Webserver started at http://127.0.98.1:45395/ using document root <none> and password file <none>
I20260925 13:12:31.202801 3510 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:31.202874 3510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:31.202998 3510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:31.203935 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "eda54aad632841be95b3d31c0c820c5c"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "34713aa9cb1f693bf60f9e7511236473"
server_key_iv: "476e8eb2b8138fc9c082c49da8f49b21"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.204222 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eda54aad632841be95b3d31c0c820c5c"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "34713aa9cb1f693bf60f9e7511236473"
server_key_iv: "476e8eb2b8138fc9c082c49da8f49b21"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.205427 3510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:31.206179 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.206355 3510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.206449 3510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "eda54aad632841be95b3d31c0c820c5c"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "34713aa9cb1f693bf60f9e7511236473"
server_key_iv: "476e8eb2b8138fc9c082c49da8f49b21"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.206538 3510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:31.240350 3510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:31.242807 3510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:31.242971 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:31.243212 3510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:31.243544 3510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:31.243602 3510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.243644 3510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:31.243672 3510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.249351 3510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37649
I20260925 13:12:31.249398 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37649 every 8 connection(s)
I20260925 13:12:31.249744 3510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:31.253616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3510
I20260925 13:12:31.253747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:31.253976 392 external_mini_cluster.cc:1428] Setting key 1e5b1083e1354311dc25b45f3b094e59
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:31.259855 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.250510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32849 (local address 127.0.98.62:45759)
0925 13:12:31.250577 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:31.250579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:31.250775 (+ 196us) server_negotiation.cc:407] Connection header received
0925 13:12:31.251300 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:31.251301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:31.251316 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:31.251378 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:31.252276 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.252799 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.254052 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.254388 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.254455 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.256860 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:31.256862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:31.256863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:31.256873 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:31.258516 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.258917 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.258918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.258919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.258976 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.259233 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.259234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.259234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.259345 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:31.259420 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:31.259686 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 13:12:31.259766 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:12:31.260041 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.250479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45759 (local address 127.0.98.1:32849)
0925 13:12:31.250649 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:31.250685 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:31.251151 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:31.251433 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:31.251436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:31.252001 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:31.252198 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:31.252203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.252875 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.252876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:31.253935 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:31.253938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.254488 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.254489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:31.254624 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.255169 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:31.255180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:31.256772 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:31.258607 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:31.258613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:31.258615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:31.258830 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:31.259066 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:31.259066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:31.259067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:31.259170 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:31.259412 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:31.259418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:31.259559 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 13:12:31.259666 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 13:12:31.259818 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
I20260925 13:12:31.260593 3639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45759
I20260925 13:12:31.260700 3639 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:31.260900 3639 heartbeater.cc:511] Master 127.0.98.62:45759 requested a full tablet report, sending...
I20260925 13:12:31.261391 3448 ts_manager.cc:194] Registered new tserver with Master: eda54aad632841be95b3d31c0c820c5c (127.0.98.1:37649)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:12:31.262066 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:31.279987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45759
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:31.358505 3644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:31.358685 3644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:31.358706 3644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:31.359732 3644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:31.359764 3644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:31.359789 3644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:31.361095 3644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:31.361258 3644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:31.361456 3644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:31.366196 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.366276 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.366294 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:31.366485 3644 server_base.cc:1034] running on GCE node
I20260925 13:12:31.366662 3644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:31.366854 3644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:31.368019 3644 hybrid_clock.cc:648] HybridClock initialized: now 1790341951368010 us; error 39 us; skew 500 ppm
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:31.370534 3644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:31.370905 3644 webserver.cc:460] Webserver started at http://127.0.98.2:42933/ using document root <none> and password file <none>
I20260925 13:12:31.371069 3644 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:31.371107 3644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:31.371205 3644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:31.372110 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c32548843fe4a1ca8cf58f61b868dfd"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "25a41661996844c0baffd00fee567238"
server_key_iv: "55e45b0301ac4406dcb18d6553df26f5"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.372398 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c32548843fe4a1ca8cf58f61b868dfd"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "25a41661996844c0baffd00fee567238"
server_key_iv: "55e45b0301ac4406dcb18d6553df26f5"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.373608 3644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:31.374399 3659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.374596 3644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:31.374677 3644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "4c32548843fe4a1ca8cf58f61b868dfd"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "25a41661996844c0baffd00fee567238"
server_key_iv: "55e45b0301ac4406dcb18d6553df26f5"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.374727 3644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:31.396759 3644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:31.398778 3644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:31.398885 3644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:31.399057 3644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:31.399362 3644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:31.399394 3644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.399415 3644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:31.399468 3644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.405261 3644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35579
I20260925 13:12:31.405308 3772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35579 every 8 connection(s)
I20260925 13:12:31.405601 3644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:31.414502 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.406370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35675 (local address 127.0.98.62:45759)
0925 13:12:31.406457 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:31.406459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:31.406646 (+ 187us) server_negotiation.cc:407] Connection header received
0925 13:12:31.407126 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:31.407127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:31.407137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:31.407193 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:31.408092 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.408577 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.409559 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.409848 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.409923 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.412077 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:31.412078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:31.412078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:31.412084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:31.413302 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.413655 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.413656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.413656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.413763 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.413986 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.413987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.413987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.414082 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:31.414130 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:31.414360 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 13:12:31.414420 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:12:31.414639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3644
I20260925 13:12:31.414726 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.406341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45759 (local address 127.0.98.2:35675)
0925 13:12:31.406531 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:31.406564 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:31.407024 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:31.407264 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:31.407267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:31.407803 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:31.407999 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:31.408003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.408647 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.408648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:31.409489 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:31.409491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.409962 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.409963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:31.410064 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.410547 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:31.410554 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:31.411984 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:31.413372 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:31.413377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:31.413378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:31.413590 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:31.413826 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:31.413827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:31.413827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:31.413925 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:31.414136 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:31.414140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:31.414253 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 13:12:31.414331 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 13:12:31.414486 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260925 13:12:31.414750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:31.414970 392 external_mini_cluster.cc:1428] Setting key 0f8e3c4bb3426eea90d5fa25c47c5812
I20260925 13:12:31.415431 3773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45759
I20260925 13:12:31.415556 3773 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:31.415782 3773 heartbeater.cc:511] Master 127.0.98.62:45759 requested a full tablet report, sending...
I20260925 13:12:31.416191 3447 ts_manager.cc:194] Registered new tserver with Master: 4c32548843fe4a1ca8cf58f61b868dfd (127.0.98.2:35579)
I20260925 13:12:31.416565 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35675
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:31.438565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45759
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:31.518180 3778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:31.518438 3778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:31.518474 3778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:31.519748 3778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:31.519805 3778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:31.519846 3778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:31.521389 3778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:31.521641 3778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:31.521888 3778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:31.526647 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.526681 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:31.526680 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:31.526904 3778 server_base.cc:1034] running on GCE node
I20260925 13:12:31.527114 3778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:31.527348 3778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:31.528517 3778 hybrid_clock.cc:648] HybridClock initialized: now 1790341951528503 us; error 23 us; skew 500 ppm
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:31.530992 3778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:31.531414 3778 webserver.cc:460] Webserver started at http://127.0.98.3:32935/ using document root <none> and password file <none>
I20260925 13:12:31.531663 3778 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:31.531731 3778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:31.531876 3778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:31.532898 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "a1bf127777fb43c19411135e4add4f4f"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "c1d0f53d4f15822e04abe9f5a4375ac4"
server_key_iv: "9478944b724cf155dd9b10f6cb07fc06"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.533236 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a1bf127777fb43c19411135e4add4f4f"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "c1d0f53d4f15822e04abe9f5a4375ac4"
server_key_iv: "9478944b724cf155dd9b10f6cb07fc06"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.534572 3778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:31.535248 3793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.535406 3778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:31.535506 3778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "a1bf127777fb43c19411135e4add4f4f"
format_stamp: "Formatted at 2026-09-25 13:12:31 on dist-test-slave-2d0s"
server_key: "c1d0f53d4f15822e04abe9f5a4375ac4"
server_key_iv: "9478944b724cf155dd9b10f6cb07fc06"
server_key_version: "encryptionkey@0"
I20260925 13:12:31.535580 3778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:31.547214 3778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:31.549322 3778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:31.549474 3778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:31.549726 3778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:31.550024 3778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:31.550077 3778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.550117 3778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:31.550143 3778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:31.555753 3778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43191
I20260925 13:12:31.555842 3906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43191 every 8 connection(s)
I20260925 13:12:31.556103 3778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:31.562701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3778
I20260925 13:12:31.562841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:31.563038 392 external_mini_cluster.cc:1428] Setting key ebfadf17653fa8042e81c3df8e1d70ee
I20260925 13:12:31.565199 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.556773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47511 (local address 127.0.98.62:45759)
0925 13:12:31.556815 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:31.556817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:31.557038 (+ 221us) server_negotiation.cc:407] Connection header received
0925 13:12:31.557462 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:31.557463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:31.557471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:31.557522 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:31.558469 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.558977 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.559967 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.560137 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.560186 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.562010 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:31.562011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:31.562012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:31.562018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:31.563719 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.564217 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.564218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.564218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.564296 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.564549 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.564550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.564551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.564685 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:31.564764 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:31.565030 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 13:12:31.565107 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:31.565366 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.556736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45759 (local address 127.0.98.3:47511)
0925 13:12:31.556896 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:31.556942 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:31.557375 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:31.557570 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:31.557573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:31.558189 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:31.558355 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:31.558360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.559058 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.559058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:31.559874 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:31.559876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.560201 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.560202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:31.560285 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.560655 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:31.560662 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:31.561926 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:31.563823 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:31.563829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:31.563831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:31.564124 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:31.564363 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:31.564364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:31.564365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:31.564478 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:31.564756 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:31.564762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:31.564892 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 13:12:31.565007 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 13:12:31.565149 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260925 13:12:31.565908 3907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45759
I20260925 13:12:31.565992 3907 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:31.566167 3907 heartbeater.cc:511] Master 127.0.98.62:45759 requested a full tablet report, sending...
I20260925 13:12:31.566501 3447 ts_manager.cc:194] Registered new tserver with Master: a1bf127777fb43c19411135e4add4f4f (127.0.98.3:43191)
I20260925 13:12:31.566838 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47511
I20260925 13:12:31.574481 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:31.595706 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:31.588860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.0.98.62:45759)
0925 13:12:31.588937 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:31.588939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:31.589053 (+ 114us) server_negotiation.cc:407] Connection header received
0925 13:12:31.589124 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:31.589125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:31.589137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:31.589204 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:31.589794 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.590259 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.590994 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:31.591188 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:31.591261 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:31.593322 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:31.593323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:31.593323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:31.593329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:31.594499 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.594835 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.594836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.594837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.594889 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:31.595082 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:31.595083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:31.595083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:31.595247 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:31.595347 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:31.595530 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 13:12:31.595606 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 13:12:31.595918 3447 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44038
I20260925 13:12:31.596306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3510
I20260925 13:12:31.600414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3644
I20260925 13:12:31.604189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3778
I20260925 13:12:31.608219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3416
2026-09-25T13:12:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1808 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3919](info): set up 2 sockets
Sep 25 13:12:31 dist-test-slave-2d0s krb5kdc[3919](info): commencing operation
krb5kdc: starting...
W20260925 13:12:32.641536 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:32 dist-test-slave-2d0s krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:32.760411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:32.840268 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:32.840474 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:32.840497 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:32.841603 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:32.841637 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:32.841651 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:32.841753 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:32.841768 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:32.843200 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:32.843386 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:32.843551 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:32.848366 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:32.848493 3932 server_base.cc:1034] running on GCE node
W20260925 13:12:32.848407 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:32.848378 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:32.848810 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:32.849193 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:32.850363 3932 hybrid_clock.cc:648] HybridClock initialized: now 1790341952850345 us; error 43 us; skew 500 ppm
Sep 25 13:12:32 dist-test-slave-2d0s krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:32.853014 3932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:32.853533 3932 webserver.cc:460] Webserver started at http://127.0.98.62:43927/ using document root <none> and password file <none>
I20260925 13:12:32.853816 3932 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:32.853889 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:32.854065 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:32.855141 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "3144548868bc435ba9a52106afdfc3f5"
format_stamp: "Formatted at 2026-09-25 13:12:32 on dist-test-slave-2d0s"
server_key: "b4f86bb1d3f80448a3e3935435da202c"
server_key_iv: "f9fe55dc373167e83244e1c5c250284c"
server_key_version: "encryptionkey@0"
I20260925 13:12:32.855535 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "3144548868bc435ba9a52106afdfc3f5"
format_stamp: "Formatted at 2026-09-25 13:12:32 on dist-test-slave-2d0s"
server_key: "b4f86bb1d3f80448a3e3935435da202c"
server_key_iv: "f9fe55dc373167e83244e1c5c250284c"
server_key_version: "encryptionkey@0"
I20260925 13:12:32.856921 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:32.857656 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:32.857883 3932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:32.857998 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "3144548868bc435ba9a52106afdfc3f5"
format_stamp: "Formatted at 2026-09-25 13:12:32 on dist-test-slave-2d0s"
server_key: "b4f86bb1d3f80448a3e3935435da202c"
server_key_iv: "f9fe55dc373167e83244e1c5c250284c"
server_key_version: "encryptionkey@0"
I20260925 13:12:32.858088 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:32.866549 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:32.869038 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:32.869205 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:32.873386 3932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33751
I20260925 13:12:32.873417 3999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33751 every 8 connection(s)
I20260925 13:12:32.873849 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:32.874715 4000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:32.875165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 3932
I20260925 13:12:32.875288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:32.875494 392 external_mini_cluster.cc:1428] Setting key 9ed2419bf9d22e6289c9b97e1ff00a06
I20260925 13:12:32.877786 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5: Bootstrap starting.
I20260925 13:12:32.878396 4000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:32.878701 4000 log.cc:824] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:32.879494 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5: No bootstrap required, opened a new log
I20260925 13:12:32.882107 4000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3144548868bc435ba9a52106afdfc3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33751 } }
I20260925 13:12:32.882247 4000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:32.882269 4000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3144548868bc435ba9a52106afdfc3f5, State: Initialized, Role: FOLLOWER
I20260925 13:12:32.882403 4000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3144548868bc435ba9a52106afdfc3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33751 } }
I20260925 13:12:32.882481 4000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:32.882507 4000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:32.882572 4000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 13:12:32 dist-test-slave-2d0s krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:32.883165 4000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3144548868bc435ba9a52106afdfc3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33751 } }
I20260925 13:12:32.883287 4000 leader_election.cc:302] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3144548868bc435ba9a52106afdfc3f5; no voters:
I20260925 13:12:32.883493 4000 leader_election.cc:288] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:32.883533 4005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:32.883656 4005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3144548868bc435ba9a52106afdfc3f5, State: Running, Role: LEADER
I20260925 13:12:32.883770 4000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:32.883836 4005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3144548868bc435ba9a52106afdfc3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33751 } }
I20260925 13:12:32.884393 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3144548868bc435ba9a52106afdfc3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3144548868bc435ba9a52106afdfc3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33751 } } }
I20260925 13:12:32.884385 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3144548868bc435ba9a52106afdfc3f5. Latest consensus state: current_term: 1 leader_uuid: "3144548868bc435ba9a52106afdfc3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3144548868bc435ba9a52106afdfc3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33751 } } }
I20260925 13:12:32.884514 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:32.884539 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:32.885262 4014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:32.885949 4014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:32.887442 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:32.876100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.0.98.62:33751)
0925 13:12:32.876280 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:32.876289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:32.876326 (+ 37us) server_negotiation.cc:407] Connection header received
0925 13:12:32.876732 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:32.876738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:32.876786 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:32.876911 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:32.877756 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:32.878825 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:32.879839 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:32.880244 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:32.880335 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:32.883257 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:32.883261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:32.883261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:32.883275 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:32.885650 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:32.886025 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:32.886028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:32.886028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:32.886136 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:32.886366 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:32.886367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:32.886367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:32.886692 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:32.886872 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:32.887063 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 13:12:32.887206 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260925 13:12:32.887925 4014 catalog_manager.cc:1347] Generated new cluster ID: 7b267e390ba049d6b78bed9f1524cb8c
I20260925 13:12:32.887981 4014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:32.901984 4014 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:32.902760 4014 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:32.911497 4014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3144548868bc435ba9a52106afdfc3f5: Generated new TSK 0
I20260925 13:12:32.911741 4014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:32.948877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33751
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:33.041915 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:33.042140 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:33.042164 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:33.043367 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:33.043411 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:33.043438 4026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:33.044978 4026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:33.045239 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:33.045485 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:33.050632 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:33.050649 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:33.050666 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:33.050966 4026 server_base.cc:1034] running on GCE node
I20260925 13:12:33.051210 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:33.051470 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:33.052624 4026 hybrid_clock.cc:648] HybridClock initialized: now 1790341953052623 us; error 39 us; skew 500 ppm
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:33.055358 4026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:33.055814 4026 webserver.cc:460] Webserver started at http://127.0.98.1:41289/ using document root <none> and password file <none>
I20260925 13:12:33.056063 4026 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:33.056133 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:33.056279 4026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:33.057366 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "26780f1b585444648437d085683e0b27"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "481baad02ccbde9766d72e463e1b2362"
server_key_iv: "6c6cdf580d92161b5c3684ca05876c00"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.057816 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26780f1b585444648437d085683e0b27"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "481baad02ccbde9766d72e463e1b2362"
server_key_iv: "6c6cdf580d92161b5c3684ca05876c00"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.059180 4026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:33.059921 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.060151 4026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:33.060258 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "26780f1b585444648437d085683e0b27"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "481baad02ccbde9766d72e463e1b2362"
server_key_iv: "6c6cdf580d92161b5c3684ca05876c00"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.060346 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:33.073189 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:33.075234 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:33.075400 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:33.075677 4026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:33.075995 4026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:33.076049 4026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.076085 4026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:33.076133 4026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.082072 4026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41561
I20260925 13:12:33.082147 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41561 every 8 connection(s)
I20260925 13:12:33.082509 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:33.084396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4026
I20260925 13:12:33.084523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:33.084717 392 external_mini_cluster.cc:1428] Setting key 623180fa06e1f4bd4cfd046c14310948
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:33.093503 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.083418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55873 (local address 127.0.98.62:33751)
0925 13:12:33.083503 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:33.083505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:33.083744 (+ 239us) server_negotiation.cc:407] Connection header received
0925 13:12:33.085019 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:33.085020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:33.085033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:33.085097 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:33.085951 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.086458 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.087482 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.087718 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.087807 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.090429 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:33.090430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:33.090430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:33.090436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:33.091924 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.092424 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.092425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.092425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.092476 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.092775 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.092775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.092775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.092874 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:33.092923 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:33.093338 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 13:12:33.093424 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:12:33.093760 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.083392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33751 (local address 127.0.98.1:55873)
0925 13:12:33.083566 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:33.083607 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:33.084899 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:33.085175 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:33.085179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:33.085571 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:33.085855 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:33.085861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.086570 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.086571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:33.087392 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:33.087394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.087817 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.087818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:33.087951 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.088561 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:33.088569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:33.090327 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:33.092015 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:33.092022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:33.092024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:33.092341 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:33.092547 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:33.092548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:33.092548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:33.092704 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:33.092945 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:33.092953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:33.093140 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 13:12:33.093297 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 13:12:33.093484 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260925 13:12:33.094405 4155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33751
I20260925 13:12:33.094506 4155 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:33.094671 4155 heartbeater.cc:511] Master 127.0.98.62:33751 requested a full tablet report, sending...
I20260925 13:12:33.095235 3964 ts_manager.cc:194] Registered new tserver with Master: 26780f1b585444648437d085683e0b27 (127.0.98.1:41561)
I20260925 13:12:33.095830 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:33.112603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33751
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:33.193756 4160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:33.193959 4160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:33.193982 4160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:33.195152 4160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:33.195196 4160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:33.195223 4160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:33.196756 4160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:33.197053 4160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:33.197317 4160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:33.202402 4166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:33.202432 4165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:33.202405 4168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:33.202672 4160 server_base.cc:1034] running on GCE node
I20260925 13:12:33.202878 4160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:33.203089 4160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:33.204253 4160 hybrid_clock.cc:648] HybridClock initialized: now 1790341953204243 us; error 39 us; skew 500 ppm
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:33.207003 4160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:33.207458 4160 webserver.cc:460] Webserver started at http://127.0.98.2:46577/ using document root <none> and password file <none>
I20260925 13:12:33.207653 4160 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:33.207693 4160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:33.207782 4160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:33.208789 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "76cf69acac254ad099e1050d0011371d"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "6a1bb8fc6df0ccda84ccb82e5cda8edc"
server_key_iv: "aec7c1ae6ba768c404b1fcd330209296"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.209085 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76cf69acac254ad099e1050d0011371d"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "6a1bb8fc6df0ccda84ccb82e5cda8edc"
server_key_iv: "aec7c1ae6ba768c404b1fcd330209296"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.210381 4160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:33.211079 4175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.211308 4160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:33.211416 4160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "76cf69acac254ad099e1050d0011371d"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "6a1bb8fc6df0ccda84ccb82e5cda8edc"
server_key_iv: "aec7c1ae6ba768c404b1fcd330209296"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.211493 4160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:33.231765 4160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:33.233835 4160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:33.233995 4160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:33.234232 4160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:33.234549 4160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:33.234581 4160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.234603 4160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:33.234647 4160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.240422 4160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39711
I20260925 13:12:33.240504 4288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39711 every 8 connection(s)
I20260925 13:12:33.240869 4160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:33.247813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4160
I20260925 13:12:33.247946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:33.248149 392 external_mini_cluster.cc:1428] Setting key 403192d647dae6f0aee6920476f0a4f6
I20260925 13:12:33.250674 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.241761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53141 (local address 127.0.98.62:33751)
0925 13:12:33.241831 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:33.241833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:33.242080 (+ 247us) server_negotiation.cc:407] Connection header received
0925 13:12:33.243023 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:33.243024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:33.243036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:33.243087 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:33.243676 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.244165 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.245142 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.245327 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.245378 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.247626 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:33.247628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:33.247628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:33.247634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:33.249186 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.249623 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.249625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.249625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.249738 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.250011 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.250012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.250012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.250132 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:33.250205 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:33.250476 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 13:12:33.250566 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:12:33.250875 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.241765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33751 (local address 127.0.98.2:53141)
0925 13:12:33.241953 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:33.241990 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:33.242929 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:33.243151 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:33.243154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:33.243410 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:33.243610 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:33.243614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.244239 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.244239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:33.245073 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:33.245075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.245399 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.245399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:33.245495 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.246050 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:33.246059 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:33.247527 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:33.249289 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:33.249296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:33.249298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:33.249526 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:33.249803 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:33.249804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:33.249805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:33.249929 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:33.250196 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:33.250203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:33.250347 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 13:12:33.250456 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 13:12:33.250620 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260925 13:12:33.251358 4289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33751
I20260925 13:12:33.251467 4289 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:33.251667 4289 heartbeater.cc:511] Master 127.0.98.62:33751 requested a full tablet report, sending...
I20260925 13:12:33.252028 3964 ts_manager.cc:194] Registered new tserver with Master: 76cf69acac254ad099e1050d0011371d (127.0.98.2:39711)
I20260925 13:12:33.252476 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53141
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:33.276029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33751
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:33.357388 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:33.357595 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:33.357618 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:33.358824 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:33.358889 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:33.358916 4294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:33.360509 4294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:33.360823 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:33.361057 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:33.366118 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:33.366099 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:33.366103 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:33.366312 4294 server_base.cc:1034] running on GCE node
I20260925 13:12:33.366513 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:33.366758 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:33.367921 4294 hybrid_clock.cc:648] HybridClock initialized: now 1790341953367903 us; error 55 us; skew 500 ppm
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:33.370409 4294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:33.370860 4294 webserver.cc:460] Webserver started at http://127.0.98.3:43415/ using document root <none> and password file <none>
I20260925 13:12:33.371095 4294 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:33.371162 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:33.371345 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:33.372399 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "737593fafef54ef48cfc59d872945f36"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "b22461a259f97b4d66fecb0f6a704351"
server_key_iv: "a3ee596dd0f6a153d02dfbcabe3aa4e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.372771 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "737593fafef54ef48cfc59d872945f36"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "b22461a259f97b4d66fecb0f6a704351"
server_key_iv: "a3ee596dd0f6a153d02dfbcabe3aa4e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.374163 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:33.374919 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.375113 4294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:33.375217 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "737593fafef54ef48cfc59d872945f36"
format_stamp: "Formatted at 2026-09-25 13:12:33 on dist-test-slave-2d0s"
server_key: "b22461a259f97b4d66fecb0f6a704351"
server_key_iv: "a3ee596dd0f6a153d02dfbcabe3aa4e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:33.375291 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:33.399601 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:33.401635 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:33.401849 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:33.402109 4294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:33.402413 4294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:33.402448 4294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.402500 4294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:33.402526 4294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:33.408180 4294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33309
I20260925 13:12:33.408221 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33309 every 8 connection(s)
I20260925 13:12:33.408535 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:12:33.410915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4294
I20260925 13:12:33.411013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:33.411207 392 external_mini_cluster.cc:1428] Setting key 980e4b8873d351674cd4e125405a697b
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:33.417846 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.409233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47871 (local address 127.0.98.62:33751)
0925 13:12:33.409304 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:33.409306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:33.409503 (+ 197us) server_negotiation.cc:407] Connection header received
0925 13:12:33.410469 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:33.410470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:33.410514 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:33.410565 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:33.411444 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.412067 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.413087 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.413295 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.413380 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.415590 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:33.415591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:33.415592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:33.415598 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:33.416717 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.417058 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.417059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.417059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.417106 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.417312 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.417312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.417313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.417407 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:33.417456 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:33.417724 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 13:12:33.417783 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:12:33.418022 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.409219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33751 (local address 127.0.98.3:47871)
0925 13:12:33.409371 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:33.409407 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:33.410379 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:33.410622 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:33.410625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:33.410984 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:33.411321 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:33.411328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.412164 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.412166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:33.412964 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:33.412966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.413388 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.413389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:33.413513 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.414019 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:33.414037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:33.415477 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:33.416780 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:33.416786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:33.416787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:33.416992 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:33.417161 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:33.417162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:33.417162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:33.417260 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:33.417457 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:33.417461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:33.417575 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 13:12:33.417653 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 13:12:33.417829 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260925 13:12:33.418467 4423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33751
I20260925 13:12:33.418571 4423 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:33.418716 4423 heartbeater.cc:511] Master 127.0.98.62:33751 requested a full tablet report, sending...
I20260925 13:12:33.419045 3964 ts_manager.cc:194] Registered new tserver with Master: 737593fafef54ef48cfc59d872945f36 (127.0.98.3:33309)
I20260925 13:12:33.419371 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47871
I20260925 13:12:33.422784 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:33.444437 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.437356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.0.98.1:41561)
0925 13:12:33.437514 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:33.437520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:33.437534 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:12:33.437572 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:33.437574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:33.437589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:33.437655 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:33.438232 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.438930 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.439641 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.439949 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.440007 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.442134 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:33.442141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:33.442142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:33.442148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:33.443519 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.443805 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.443808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.443809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.443859 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.443997 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.443997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.443997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.444113 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:33.444213 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:33.444264 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:12:33.444322 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:33.451902 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:33.445770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.98.62:33751)
0925 13:12:33.445811 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:33.445813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:33.445857 (+ 44us) server_negotiation.cc:407] Connection header received
0925 13:12:33.445908 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:33.445909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:33.445917 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:33.445959 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:33.446342 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.446791 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.447432 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:33.447666 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:33.447719 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:33.449830 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:33.449831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:33.449832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:33.449838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:33.450972 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.451284 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.451285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.451286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.451332 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:33.451472 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:33.451472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:33.451473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:33.451593 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:33.451660 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:33.451741 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:12:33.451800 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 13:12:33.452174 3964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55706
I20260925 13:12:33.452561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4026
I20260925 13:12:33.457391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4160
I20260925 13:12:33.462589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4294
I20260925 13:12:33.466670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 3932
2026-09-25T13:12:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1858 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[4435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[4435](info): set up 2 sockets
Sep 25 13:12:33 dist-test-slave-2d0s krb5kdc[4435](info): commencing operation
krb5kdc: starting...
W20260925 13:12:34.493803 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:34 dist-test-slave-2d0s krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:34.608114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:34.687255 4448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:34.687448 4448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:34.687470 4448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:34.688606 4448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:34.688645 4448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:34.688664 4448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:34.688676 4448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:34.688726 4448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:34.690231 4448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:34.690485 4448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:34.690729 4448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:34.695443 4453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:34.695675 4448 server_base.cc:1034] running on GCE node
W20260925 13:12:34.695451 4454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:34.695480 4456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:34.695999 4448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:34.696241 4448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:34.697420 4448 hybrid_clock.cc:648] HybridClock initialized: now 1790341954697397 us; error 39 us; skew 500 ppm
Sep 25 13:12:34 dist-test-slave-2d0s krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:34.699775 4448 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:34.700112 4448 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:34.700160 4448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:34.700248 4448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:34.701200 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "70f2e5821ad04a22b11f839c146ca041"
format_stamp: "Formatted at 2026-09-25 13:12:34 on dist-test-slave-2d0s"
server_key: "8374a408e0bfff8d5c6be0613a77d7d1"
server_key_iv: "35f212a7dd81120a0207854ddf7b711f"
server_key_version: "encryptionkey@0"
I20260925 13:12:34.701489 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "70f2e5821ad04a22b11f839c146ca041"
format_stamp: "Formatted at 2026-09-25 13:12:34 on dist-test-slave-2d0s"
server_key: "8374a408e0bfff8d5c6be0613a77d7d1"
server_key_iv: "35f212a7dd81120a0207854ddf7b711f"
server_key_version: "encryptionkey@0"
I20260925 13:12:34.702791 4448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 13:12:34.703531 4462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:34.703713 4448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:34.703783 4448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "70f2e5821ad04a22b11f839c146ca041"
format_stamp: "Formatted at 2026-09-25 13:12:34 on dist-test-slave-2d0s"
server_key: "8374a408e0bfff8d5c6be0613a77d7d1"
server_key_iv: "35f212a7dd81120a0207854ddf7b711f"
server_key_version: "encryptionkey@0"
I20260925 13:12:34.703833 4448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:34.718722 4448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:34.720504 4448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:34.720606 4448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:34.724547 4448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46487
I20260925 13:12:34.724608 4514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46487 every 8 connection(s)
I20260925 13:12:34.724849 4448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:34.725802 4515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:34.727931 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: Bootstrap starting.
I20260925 13:12:34.728443 4515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:34.728677 4515 log.cc:824] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:34.729247 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: No bootstrap required, opened a new log
I20260925 13:12:34.730697 4515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f2e5821ad04a22b11f839c146ca041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260925 13:12:34.730823 4515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:34.730844 4515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f2e5821ad04a22b11f839c146ca041, State: Initialized, Role: FOLLOWER
I20260925 13:12:34.730937 4515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f2e5821ad04a22b11f839c146ca041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260925 13:12:34.730989 4515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:34.731009 4515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:34.731038 4515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:34.731774 4515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f2e5821ad04a22b11f839c146ca041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260925 13:12:34.731853 4515 leader_election.cc:302] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f2e5821ad04a22b11f839c146ca041; no voters:
I20260925 13:12:34.731978 4515 leader_election.cc:288] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:34.732025 4518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:34.732204 4518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [term 1 LEADER]: Becoming Leader. State: Replica: 70f2e5821ad04a22b11f839c146ca041, State: Running, Role: LEADER
I20260925 13:12:34.732220 4515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:34.732311 4518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f2e5821ad04a22b11f839c146ca041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260925 13:12:34.732652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4448
I20260925 13:12:34.732769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:34.733000 392 external_mini_cluster.cc:1428] Setting key a95e8e22ca95d5a77641ca4b105dfdfb
I20260925 13:12:34.733626 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f2e5821ad04a22b11f839c146ca041. Latest consensus state: current_term: 1 leader_uuid: "70f2e5821ad04a22b11f839c146ca041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f2e5821ad04a22b11f839c146ca041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } } }
I20260925 13:12:34.734081 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:34.733768 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f2e5821ad04a22b11f839c146ca041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f2e5821ad04a22b11f839c146ca041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } } }
W20260925 13:12:34.734176 4535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 13:12:34.734213 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041 [sys.catalog]: This master's current role is: LEADER
W20260925 13:12:34.734215 4535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:34.734333 4537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:34.734500 4537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:34.735926 4537 catalog_manager.cc:1347] Generated new cluster ID: 81f9ad30c71a4e6382b5e6eae3fd6870
I20260925 13:12:34.735988 4537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:34 dist-test-slave-2d0s krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:34.743402 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:34.733443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.0.98.62:46487)
0925 13:12:34.733823 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:34.733830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:34.733875 (+ 45us) server_negotiation.cc:407] Connection header received
0925 13:12:34.733989 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:34.733997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:34.734061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:34.734216 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:34.734959 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:34.736219 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:34.736980 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:34.737409 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:34.737525 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:34.739971 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:34.739974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:34.739975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:34.739987 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:34.741824 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:34.742226 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:34.742229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:34.742229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:34.742326 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:34.742557 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:34.742558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:34.742559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:34.742792 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:34.742989 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:34.743056 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 13:12:34.743194 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":63,"threads_started":1}
I20260925 13:12:34.762044 4537 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:34.762593 4537 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:34.772284 4537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: Generated new TSK 0
I20260925 13:12:34.772469 4537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:34.798849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:34.877216 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:34.877424 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:34.877462 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:34.878651 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:34.878717 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:34.878762 4541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:34.880223 4541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:34.880484 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:34.880726 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:34.885483 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:34.885565 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:34.885481 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:34.885805 4541 server_base.cc:1034] running on GCE node
I20260925 13:12:34.886004 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:34.886178 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:34.887347 4541 hybrid_clock.cc:648] HybridClock initialized: now 1790341954887325 us; error 41 us; skew 500 ppm
Sep 25 13:12:34 dist-test-slave-2d0s krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:34.889865 4541 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:34.890182 4541 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:34.890251 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:34.890413 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:34.891434 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "33c9cc7bb62b4d1ba58ffe8512680310"
format_stamp: "Formatted at 2026-09-25 13:12:34 on dist-test-slave-2d0s"
server_key: "e4c08988d339aedcd9b62571dd50c0ec"
server_key_iv: "7e174473cba209bd562b17253d89e5eb"
server_key_version: "encryptionkey@0"
I20260925 13:12:34.891804 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "33c9cc7bb62b4d1ba58ffe8512680310"
format_stamp: "Formatted at 2026-09-25 13:12:34 on dist-test-slave-2d0s"
server_key: "e4c08988d339aedcd9b62571dd50c0ec"
server_key_iv: "7e174473cba209bd562b17253d89e5eb"
server_key_version: "encryptionkey@0"
I20260925 13:12:34.892994 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:34.893729 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:34.893898 4541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:34.893983 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "33c9cc7bb62b4d1ba58ffe8512680310"
format_stamp: "Formatted at 2026-09-25 13:12:34 on dist-test-slave-2d0s"
server_key: "e4c08988d339aedcd9b62571dd50c0ec"
server_key_iv: "7e174473cba209bd562b17253d89e5eb"
server_key_version: "encryptionkey@0"
I20260925 13:12:34.894066 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:34.904330 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:34.906127 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:34.906355 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:34.906585 4541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:34.906872 4541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:34.906924 4541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:34.906962 4541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:34.906992 4541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:34.912626 4541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39449
I20260925 13:12:34.912668 4668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39449 every 8 connection(s)
I20260925 13:12:34.912900 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:34.913301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4541
I20260925 13:12:34.913405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:34.913605 392 external_mini_cluster.cc:1428] Setting key ceeaa3a2f91384f6f39c0f5bf77aeac6
Sep 25 13:12:34 dist-test-slave-2d0s krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:12:34.922149 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:34.913579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43035 (local address 127.0.98.62:46487)
0925 13:12:34.913625 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:34.913628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:34.913851 (+ 223us) server_negotiation.cc:407] Connection header received
0925 13:12:34.914501 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:34.914502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:34.914515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:34.914561 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:34.915432 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:34.915931 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:34.916945 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:34.917176 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:34.917264 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:34.919291 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:34.919292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:34.919293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:34.919302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:34.920743 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:34.921156 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:34.921157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:34.921158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:34.921218 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:34.921461 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:34.921462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:34.921463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:34.921580 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:34.921653 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:34.921936 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 13:12:34.922061 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:34.922278 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:34.913565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.1:43035)
0925 13:12:34.913725 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:34.913766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:34.914387 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:34.914622 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:34.914625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:34.914874 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:34.915326 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:34.915330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:34.916045 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:34.916046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:34.916848 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:34.916850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:34.917273 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:34.917274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:34.917403 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:34.917916 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:34.917924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:34.919194 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:34.920844 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:34.920851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:34.920853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:34.921096 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:34.921284 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:34.921285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:34.921286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:34.921392 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:34.921656 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:34.921678 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:34.921807 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 13:12:34.921914 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 13:12:34.922068 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":102,"threads_started":1}
I20260925 13:12:34.922771 4669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46487
I20260925 13:12:34.922854 4669 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:34.923030 4669 heartbeater.cc:511] Master 127.0.98.62:46487 requested a full tablet report, sending...
I20260925 13:12:34.923547 4479 ts_manager.cc:194] Registered new tserver with Master: 33c9cc7bb62b4d1ba58ffe8512680310 (127.0.98.1:39449)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:12:34.924377 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:34.939966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:35.019874 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:35.020068 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:35.020092 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:35.021111 4674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:35.021148 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:35.021175 4674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:35.022531 4674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:35.022698 4674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:35.022858 4674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:35.027722 4679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.027849 4674 server_base.cc:1034] running on GCE node
W20260925 13:12:35.027829 4680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.027830 4682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.028174 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:35.028426 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:35.029577 4674 hybrid_clock.cc:648] HybridClock initialized: now 1790341955029574 us; error 40 us; skew 500 ppm
Sep 25 13:12:35 dist-test-slave-2d0s krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:35.031987 4674 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:35.032300 4674 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:35.032385 4674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:35.032539 4674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:35.033573 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "47465c22c55040cfba7b6f0f8a466690"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "2d19144c995e8cb18e95a1ca2092b6b0"
server_key_iv: "64c368f16445769bc5fab7391b4bb2c6"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.034006 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "47465c22c55040cfba7b6f0f8a466690"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "2d19144c995e8cb18e95a1ca2092b6b0"
server_key_iv: "64c368f16445769bc5fab7391b4bb2c6"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.035226 4674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:35.035969 4688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.036134 4674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:35.036226 4674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "47465c22c55040cfba7b6f0f8a466690"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "2d19144c995e8cb18e95a1ca2092b6b0"
server_key_iv: "64c368f16445769bc5fab7391b4bb2c6"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.036309 4674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:35.054944 4674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:35.056612 4674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:35.056758 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:35.056959 4674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:35.057245 4674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:35.057296 4674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.057335 4674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:35.057363 4674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.062911 4674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38847
I20260925 13:12:35.062948 4801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38847 every 8 connection(s)
I20260925 13:12:35.063177 4674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:12:35.064294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4674
I20260925 13:12:35.064396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:35.064599 392 external_mini_cluster.cc:1428] Setting key 07333e66b374a69ba4bf8be00ab89c9a
Sep 25 13:12:35 dist-test-slave-2d0s krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:35.072544 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.063837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40371 (local address 127.0.98.62:46487)
0925 13:12:35.063916 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.063918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.064074 (+ 156us) server_negotiation.cc:407] Connection header received
0925 13:12:35.064584 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.064585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.064610 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.064663 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.065625 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.066209 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.067209 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.067523 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.067593 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.069604 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.069605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.069606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:35.069614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.071223 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:35.071607 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:35.071609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:35.071610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:35.071668 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:35.071891 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:35.071892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:35.071892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:35.072061 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.072140 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.072351 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.072445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:12:35.072582 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.063784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.2:40371)
0925 13:12:35.063928 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.063968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.064466 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.064723 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.064726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.065010 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.065536 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.065541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.066312 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.066313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.067105 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.067106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.067605 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.067606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.067707 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.068192 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:35.068200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.069516 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.071324 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:35.071329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:35.071330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:35.071531 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:35.071738 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:35.071739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:35.071739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:35.071823 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:35.072130 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.072133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.072232 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.072306 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.072410 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":85,"threads_started":1}
I20260925 13:12:35.073051 4802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46487
I20260925 13:12:35.073144 4802 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:35.073316 4802 heartbeater.cc:511] Master 127.0.98.62:46487 requested a full tablet report, sending...
I20260925 13:12:35.073797 4479 ts_manager.cc:194] Registered new tserver with Master: 47465c22c55040cfba7b6f0f8a466690 (127.0.98.2:38847)
I20260925 13:12:35.074183 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:35.088791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:35.165439 4807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:35.165676 4807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:35.165735 4807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:35.166909 4807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:35.166991 4807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:35.167044 4807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:35.168553 4807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:35.168819 4807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:35.169097 4807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:35.173903 4815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.173939 4813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.173915 4812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.174225 4807 server_base.cc:1034] running on GCE node
I20260925 13:12:35.174438 4807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:35.174696 4807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:35.175854 4807 hybrid_clock.cc:648] HybridClock initialized: now 1790341955175846 us; error 60 us; skew 500 ppm
Sep 25 13:12:35 dist-test-slave-2d0s krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:35.178328 4807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:35.178678 4807 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:35.178752 4807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:35.178896 4807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:35.179915 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "c14d493ed7ce4fdc98fc5c235634db1b"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "9ecf1f6c23fd9f498a73c27522a13b1a"
server_key_iv: "536997506735a27895cf35d7e9358311"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.180294 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c14d493ed7ce4fdc98fc5c235634db1b"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "9ecf1f6c23fd9f498a73c27522a13b1a"
server_key_iv: "536997506735a27895cf35d7e9358311"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.181582 4807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:35.182363 4821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.182590 4807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:35.182683 4807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "c14d493ed7ce4fdc98fc5c235634db1b"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "9ecf1f6c23fd9f498a73c27522a13b1a"
server_key_iv: "536997506735a27895cf35d7e9358311"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.182767 4807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:35.190354 4807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:35.192240 4807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:35.192399 4807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:35.192636 4807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:35.192926 4807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:35.192977 4807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.193012 4807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:35.193038 4807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.198726 4807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45619
I20260925 13:12:35.198807 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45619 every 8 connection(s)
I20260925 13:12:35.199132 4807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:12:35.202950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4807
I20260925 13:12:35.203045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:35.203255 392 external_mini_cluster.cc:1428] Setting key b4e5354609d7b563a059e85f088b1130
Sep 25 13:12:35 dist-test-slave-2d0s krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:35.208935 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.199884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40735 (local address 127.0.98.62:46487)
0925 13:12:35.199964 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.199966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.200174 (+ 208us) server_negotiation.cc:407] Connection header received
0925 13:12:35.200697 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.200699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.200710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.200775 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.201815 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.202321 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.203398 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.203652 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.203748 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.206099 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.206100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.206101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:35.206109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.207537 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:35.207938 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:35.207940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:35.207940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:35.208011 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:35.208263 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:35.208264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:35.208264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:35.208389 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.208466 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.208760 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.208842 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 13:12:35.209154 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.199849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.3:40735)
0925 13:12:35.200048 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.200085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.200583 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.200833 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.200836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.201103 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.201601 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.201608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.202401 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.202402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.203325 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.203328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.203758 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.203760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.203938 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.204514 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:35.204522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.206008 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.207628 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:35.207635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:35.207637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:35.207861 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:35.208080 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:35.208081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:35.208082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:35.208193 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:35.208457 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.208464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.208619 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.208726 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.208890 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260925 13:12:35.209652 4935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46487
I20260925 13:12:35.209792 4935 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:35.209918 4935 heartbeater.cc:511] Master 127.0.98.62:46487 requested a full tablet report, sending...
I20260925 13:12:35.210314 4479 ts_manager.cc:194] Registered new tserver with Master: c14d493ed7ce4fdc98fc5c235634db1b (127.0.98.3:45619)
I20260925 13:12:35.210739 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40735
I20260925 13:12:35.215026 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:12:35.222273 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.216379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.0.98.62:46487)
0925 13:12:35.216439 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.216441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.216530 (+ 89us) server_negotiation.cc:407] Connection header received
0925 13:12:35.216571 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.216572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.216580 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.216629 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.217196 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.217731 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.218453 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.218672 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.218727 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.219879 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.219880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.219880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:35.219886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.221090 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:35.221418 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:35.221419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:35.221419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:35.221494 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:35.221722 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:35.221723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:35.221723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:35.221900 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.222011 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.222143 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.222207 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 13:12:35.223887 4479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:35.225432 4479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:35.232084 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.228248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.0.98.1:39449)
0925 13:12:35.228437 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.228442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.228450 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:35.228490 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.228492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.228508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.228725 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:35.228800 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.229860 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.230948 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.231678 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231793 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.231854 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.231956 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.231992 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260925 13:12:35.232100 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.227820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38847 (local address 127.0.0.1:45892)
0925 13:12:35.227991 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.228052 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.228122 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.228399 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.228403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.228405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:35.228484 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.228486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.229905 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.229906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.230966 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.230968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.231802 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.231888 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.231892 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.231928 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.231980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260925 13:12:35.232102 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.228030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39449 (local address 127.0.0.1:35150)
0925 13:12:35.228176 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.228186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.228239 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.228617 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.228618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.228620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:35.228655 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.228657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.229970 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.229971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.230863 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.230866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.231812 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.231876 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.231879 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.231930 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.231974 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260925 13:12:35.232183 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.227939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.98.3:45619)
0925 13:12:35.228372 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.228376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.228386 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:35.228425 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.228427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.228443 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.228698 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:35.228772 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.229546 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231069 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.231563 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231652 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.231712 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.232047 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.232095 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":77,"threads_started":1}
I20260925 13:12:35.232215 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.227846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45619 (local address 127.0.0.1:51880)
0925 13:12:35.228145 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.228154 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.228197 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.228555 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.228556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.228558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:35.228621 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.228623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.229646 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.229647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.230969 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.230971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.231663 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.231741 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.231754 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.231865 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.231930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260925 13:12:35.232625 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.227918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.0.98.2:38847)
0925 13:12:35.228108 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.228113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.228124 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:35.228258 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.228261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.228276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.228575 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:35.228662 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.229805 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.231041 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.231711 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.232380 (+ 669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.232476 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.232499 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.232545 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260925 13:12:35.232952 4869 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe12982aff4c2c997f5532ebeea112 (DEFAULT_TABLE table=test-table [id=f58070759cfb4e6c9f7798e6e876da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:35.233372 4869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe12982aff4c2c997f5532ebeea112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.233381 4603 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe12982aff4c2c997f5532ebeea112 (DEFAULT_TABLE table=test-table [id=f58070759cfb4e6c9f7798e6e876da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:35.233724 4603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe12982aff4c2c997f5532ebeea112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.234292 4736 tablet_service.cc:1467] Processing CreateTablet for tablet a7fe12982aff4c2c997f5532ebeea112 (DEFAULT_TABLE table=test-table [id=f58070759cfb4e6c9f7798e6e876da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:35.234576 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fe12982aff4c2c997f5532ebeea112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.236941 4950 tablet_bootstrap.cc:492] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Bootstrap starting.
I20260925 13:12:35.237417 4949 tablet_bootstrap.cc:492] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310: Bootstrap starting.
I20260925 13:12:35.237622 4950 tablet_bootstrap.cc:654] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.238101 4949 tablet_bootstrap.cc:654] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.238286 4950 log.cc:824] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.238683 4949 log.cc:824] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.239056 4951 tablet_bootstrap.cc:492] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Bootstrap starting.
I20260925 13:12:35.239171 4950 tablet_bootstrap.cc:492] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: No bootstrap required, opened a new log
I20260925 13:12:35.239231 4950 ts_tablet_manager.cc:1397] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:12:35.239658 4951 tablet_bootstrap.cc:654] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.240213 4951 log.cc:824] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.240875 4951 tablet_bootstrap.cc:492] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: No bootstrap required, opened a new log
I20260925 13:12:35.240919 4949 tablet_bootstrap.cc:492] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310: No bootstrap required, opened a new log
I20260925 13:12:35.240978 4951 ts_tablet_manager.cc:1397] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:35.241008 4949 ts_tablet_manager.cc:1397] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 13:12:35.241425 4950 raft_consensus.cc:348] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.241609 4950 raft_consensus.cc:374] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.241644 4950 raft_consensus.cc:729] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14d493ed7ce4fdc98fc5c235634db1b, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.241815 4950 consensus_queue.cc:260] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.242101 4950 ts_tablet_manager.cc:1428] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 13:12:35.242249 4935 heartbeater.cc:503] Master 127.0.98.62:46487 was elected leader, sending a full tablet report...
I20260925 13:12:35.242827 4951 raft_consensus.cc:348] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.242998 4951 raft_consensus.cc:374] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.243052 4951 raft_consensus.cc:729] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47465c22c55040cfba7b6f0f8a466690, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.242924 4949 raft_consensus.cc:348] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.243112 4949 raft_consensus.cc:374] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.243153 4949 raft_consensus.cc:729] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c9cc7bb62b4d1ba58ffe8512680310, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.243201 4951 consensus_queue.cc:260] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.243489 4802 heartbeater.cc:503] Master 127.0.98.62:46487 was elected leader, sending a full tablet report...
I20260925 13:12:35.243278 4949 consensus_queue.cc:260] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.243494 4951 ts_tablet_manager.cc:1428] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:12:35.243745 4669 heartbeater.cc:503] Master 127.0.98.62:46487 was elected leader, sending a full tablet report...
I20260925 13:12:35.243985 4949 ts_tablet_manager.cc:1428] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:12:35.302469 4956 raft_consensus.cc:482] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:35.302603 4956 raft_consensus.cc:504] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.303054 4956 leader_election.cc:288] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c9cc7bb62b4d1ba58ffe8512680310 (127.0.98.1:39449), c14d493ed7ce4fdc98fc5c235634db1b (127.0.98.3:45619)
I20260925 13:12:35.307157 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.303294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44479 (local address 127.0.98.1:39449)
0925 13:12:35.303359 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.303361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.303371 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:35.303401 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.303402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.303414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.303513 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:35.303741 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.304604 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.306178 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.306761 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.306838 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.306866 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.307024 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.307071 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:12:35.307157 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.303323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39245 (local address 127.0.98.3:45619)
0925 13:12:35.303385 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.303387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.303552 (+ 165us) server_negotiation.cc:407] Connection header received
0925 13:12:35.303589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.303590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.303601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.303642 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:35.303899 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.304593 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.306176 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.306768 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.306838 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.306866 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.307014 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.307065 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:12:35.307160 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.303205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39449 (local address 127.0.98.2:44479)
0925 13:12:35.303284 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.303297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.303342 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.303545 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.303546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.303548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:35.303649 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.303652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.304682 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.304683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.306081 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.306085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.306848 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.306849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.306930 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.306934 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.306990 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.307043 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":76}
I20260925 13:12:35.307166 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.303251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45619 (local address 127.0.98.2:39245)
0925 13:12:35.303436 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.303443 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.303506 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.303662 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.303663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.303664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:35.303728 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.303730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.304682 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.304683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.306081 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:35.306085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.306848 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.306849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:35.306929 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.306934 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.306990 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.307043 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":51,"thread_start_us":106,"threads_started":1}
I20260925 13:12:35.307528 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe12982aff4c2c997f5532ebeea112" candidate_uuid: "47465c22c55040cfba7b6f0f8a466690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" is_pre_election: true
I20260925 13:12:35.307667 4623 raft_consensus.cc:2454] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47465c22c55040cfba7b6f0f8a466690 in term 0.
I20260925 13:12:35.307936 4690 leader_election.cc:302] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c9cc7bb62b4d1ba58ffe8512680310, 47465c22c55040cfba7b6f0f8a466690; no voters:
I20260925 13:12:35.307971 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe12982aff4c2c997f5532ebeea112" candidate_uuid: "47465c22c55040cfba7b6f0f8a466690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" is_pre_election: true
I20260925 13:12:35.308117 4889 raft_consensus.cc:2454] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47465c22c55040cfba7b6f0f8a466690 in term 0.
I20260925 13:12:35.308229 4956 raft_consensus.cc:2781] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:35.308324 4956 raft_consensus.cc:482] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:35.308373 4956 raft_consensus.cc:3037] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.309130 4956 raft_consensus.cc:504] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.309278 4956 leader_election.cc:288] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [CANDIDATE]: Term 1 election: Requested vote from peers 33c9cc7bb62b4d1ba58ffe8512680310 (127.0.98.1:39449), c14d493ed7ce4fdc98fc5c235634db1b (127.0.98.3:45619)
I20260925 13:12:35.309422 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe12982aff4c2c997f5532ebeea112" candidate_uuid: "47465c22c55040cfba7b6f0f8a466690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c14d493ed7ce4fdc98fc5c235634db1b"
I20260925 13:12:35.309437 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fe12982aff4c2c997f5532ebeea112" candidate_uuid: "47465c22c55040cfba7b6f0f8a466690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c9cc7bb62b4d1ba58ffe8512680310"
I20260925 13:12:35.309530 4889 raft_consensus.cc:3037] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.309523 4623 raft_consensus.cc:3037] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.310259 4889 raft_consensus.cc:2454] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47465c22c55040cfba7b6f0f8a466690 in term 1.
I20260925 13:12:35.310333 4623 raft_consensus.cc:2454] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47465c22c55040cfba7b6f0f8a466690 in term 1.
I20260925 13:12:35.310550 4691 leader_election.cc:302] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47465c22c55040cfba7b6f0f8a466690, c14d493ed7ce4fdc98fc5c235634db1b; no voters:
I20260925 13:12:35.310660 4956 raft_consensus.cc:2781] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:35.310829 4956 raft_consensus.cc:686] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 1 LEADER]: Becoming Leader. State: Replica: 47465c22c55040cfba7b6f0f8a466690, State: Running, Role: LEADER
I20260925 13:12:35.310938 4956 consensus_queue.cc:237] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } }
I20260925 13:12:35.311880 4478 catalog_manager.cc:5495] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47465c22c55040cfba7b6f0f8a466690 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "47465c22c55040cfba7b6f0f8a466690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47465c22c55040cfba7b6f0f8a466690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } health_report { overall_health: UNKNOWN } } }
W20260925 13:12:35.313725 4803 tablet.cc:2369] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:35.324065 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.321526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.0.98.2:38847)
0925 13:12:35.321593 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.321595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.321604 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:35.321729 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.321730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.321770 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.321838 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:35.321988 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.322471 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.323215 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:35.323404 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:35.323473 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:35.323683 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:35.323788 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:35.323854 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.323895 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.323913 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:12:35.326318 4623 raft_consensus.cc:1261] T a7fe12982aff4c2c997f5532ebeea112 P 33c9cc7bb62b4d1ba58ffe8512680310 [term 1 FOLLOWER]: Refusing update from remote peer 47465c22c55040cfba7b6f0f8a466690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:35.326318 4889 raft_consensus.cc:1261] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 1 FOLLOWER]: Refusing update from remote peer 47465c22c55040cfba7b6f0f8a466690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:35.326715 4959 consensus_queue.cc:1035] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c9cc7bb62b4d1ba58ffe8512680310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:35.326858 4956 consensus_queue.cc:1035] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c14d493ed7ce4fdc98fc5c235634db1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:35.329250 4963 mvcc.cc:205] Tried to move back new op lower bound from 7333240649013321728 to 7333240648954466304. Current Snapshot: MvccSnapshot[applied={T|T < 7333240649013321728}]
I20260925 13:12:35.329751 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333240649013321728 to 7333240648954466304. Current Snapshot: MvccSnapshot[applied={T|T < 7333240649013321728}]
I20260925 13:12:35.331802 4478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:35.331900 4478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:35.332684 4478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70f2e5821ad04a22b11f839c146ca041: Sending DeleteTablet for 3 replicas of tablet a7fe12982aff4c2c997f5532ebeea112
I20260925 13:12:35.332990 4736 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe12982aff4c2c997f5532ebeea112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45892
I20260925 13:12:35.333051 4869 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe12982aff4c2c997f5532ebeea112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51880
I20260925 13:12:35.333314 4972 tablet_replica.cc:330] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: stopping tablet replica
I20260925 13:12:35.333364 4603 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fe12982aff4c2c997f5532ebeea112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35150
I20260925 13:12:35.333406 4972 raft_consensus.cc:2229] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:35.333408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4541
I20260925 13:12:35.333537 4972 raft_consensus.cc:2258] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:35.333701 4973 tablet_replica.cc:330] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: stopping tablet replica
I20260925 13:12:35.333822 4973 raft_consensus.cc:2229] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:35.334043 4973 raft_consensus.cc:2258] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:35.334462 4973 ts_tablet_manager.cc:1905] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:35.334566 4972 ts_tablet_manager.cc:1905] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:35.336334 4972 ts_tablet_manager.cc:1918] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:35.336405 4972 log.cc:1196] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/a7fe12982aff4c2c997f5532ebeea112
I20260925 13:12:35.336403 4973 ts_tablet_manager.cc:1918] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:35.336465 4973 log.cc:1196] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/a7fe12982aff4c2c997f5532ebeea112
I20260925 13:12:35.336704 4972 ts_tablet_manager.cc:1939] T a7fe12982aff4c2c997f5532ebeea112 P 47465c22c55040cfba7b6f0f8a466690: Deleting consensus metadata
I20260925 13:12:35.336756 4973 ts_tablet_manager.cc:1939] T a7fe12982aff4c2c997f5532ebeea112 P c14d493ed7ce4fdc98fc5c235634db1b: Deleting consensus metadata
I20260925 13:12:35.337188 4464 catalog_manager.cc:4841] TS 47465c22c55040cfba7b6f0f8a466690 (127.0.98.2:38847): tablet a7fe12982aff4c2c997f5532ebeea112 (table test-table [id=f58070759cfb4e6c9f7798e6e876da57]) successfully deleted
I20260925 13:12:35.337198 4465 catalog_manager.cc:4841] TS c14d493ed7ce4fdc98fc5c235634db1b (127.0.98.3:45619): tablet a7fe12982aff4c2c997f5532ebeea112 (table test-table [id=f58070759cfb4e6c9f7798e6e876da57]) successfully deleted
W20260925 13:12:35.339977 4464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39449 (error 108)
I20260925 13:12:35.340178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4674
I20260925 13:12:35.341075 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.340914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39449 (local address 127.0.0.1:35166)
0925 13:12:35.340986 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.341033 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 13:12:35.341225 4464 catalog_manager.cc:4584] TS 33c9cc7bb62b4d1ba58ffe8512680310 (127.0.98.1:39449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7fe12982aff4c2c997f5532ebeea112: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39449: connect: Connection refused (error 111)
I20260925 13:12:35.345435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4807
I20260925 13:12:35.350557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4448
2026-09-25T13:12:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1884 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T13:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:35Z Disabled control of system clock
I20260925 13:12:35.378239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41667
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:12:35.458194 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:35.458420 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:35.458441 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:35.459553 4978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:35.459586 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:35.459599 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:35.459612 4978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:35.459625 4978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:35.460875 4978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:35.461066 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:35.461303 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:35.466075 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.466120 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.466115 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.466492 4978 server_base.cc:1034] running on GCE node
I20260925 13:12:35.466747 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:35.467089 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:35.468303 4978 hybrid_clock.cc:648] HybridClock initialized: now 1790341955468265 us; error 53 us; skew 500 ppm
I20260925 13:12:35.470002 4978 webserver.cc:460] Webserver started at http://127.0.98.62:36895/ using document root <none> and password file <none>
I20260925 13:12:35.470284 4978 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:35.470360 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:35.470525 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:35.471824 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "a6910ac093f74037977903ca816fb60f"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "b7e894f28ba2b31fb4b45fbbfd79d39a"
server_key_iv: "6a86ea2d8acc02a67bd600c85b101322"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.472234 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6910ac093f74037977903ca816fb60f"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "b7e894f28ba2b31fb4b45fbbfd79d39a"
server_key_iv: "6a86ea2d8acc02a67bd600c85b101322"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.473753 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 13:12:35.474612 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.474802 4978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:35.474907 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "a6910ac093f74037977903ca816fb60f"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "b7e894f28ba2b31fb4b45fbbfd79d39a"
server_key_iv: "6a86ea2d8acc02a67bd600c85b101322"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.475003 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:35.478674 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:35.478982 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:35.479138 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:35.483260 4978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41667
I20260925 13:12:35.483279 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41667 every 8 connection(s)
I20260925 13:12:35.483641 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:35.484282 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.487110 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f: Bootstrap starting.
I20260925 13:12:35.487718 5045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.488021 5045 log.cc:824] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.488710 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f: No bootstrap required, opened a new log
I20260925 13:12:35.490360 5045 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6910ac093f74037977903ca816fb60f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260925 13:12:35.490499 5045 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.490521 5045 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6910ac093f74037977903ca816fb60f, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.490711 5045 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6910ac093f74037977903ca816fb60f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260925 13:12:35.490773 5045 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:35.490818 5045 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:35.490867 5045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.491405 5045 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6910ac093f74037977903ca816fb60f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260925 13:12:35.491518 5045 leader_election.cc:302] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6910ac093f74037977903ca816fb60f; no voters:
I20260925 13:12:35.491698 5045 leader_election.cc:288] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:35.491743 5048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:35.491954 5045 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:35.491976 5048 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [term 1 LEADER]: Becoming Leader. State: Replica: a6910ac093f74037977903ca816fb60f, State: Running, Role: LEADER
I20260925 13:12:35.492061 5048 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6910ac093f74037977903ca816fb60f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260925 13:12:35.492314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 4978
I20260925 13:12:35.492431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:35.492475 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6910ac093f74037977903ca816fb60f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6910ac093f74037977903ca816fb60f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } } }
I20260925 13:12:35.492553 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:35.492624 392 external_mini_cluster.cc:1428] Setting key 9dc2bed8a18899359e9e7591d753f9b0
I20260925 13:12:35.492725 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6910ac093f74037977903ca816fb60f. Latest consensus state: current_term: 1 leader_uuid: "a6910ac093f74037977903ca816fb60f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6910ac093f74037977903ca816fb60f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } } }
I20260925 13:12:35.492746 5052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:35.492792 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:35.493427 5052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:35.495203 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.493311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.0.98.62:41667)
0925 13:12:35.493520 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.493529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.493560 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:12:35.493701 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.493708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.493751 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.493896 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.494599 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.494603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.494603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.494607 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.494654 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.494671 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.494776 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.494977 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.495036 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260925 13:12:35.498087 5052 catalog_manager.cc:1347] Generated new cluster ID: e35f3afce4a94f77b2c0c8ed031392a3
I20260925 13:12:35.498152 5052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:35.508714 5052 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:35.508914 5052 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:35.519025 5052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f: Generated new TSK 0
I20260925 13:12:35.519297 5052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:12:35.520696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--builtin_ntp_servers=127.0.98.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:12:35.611207 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:35.611479 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:35.611505 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:35.612717 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:35.612761 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:35.612788 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:35.614122 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:35.614362 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:35.614688 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:35.620585 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.620781 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.620599 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.620913 5069 server_base.cc:1034] running on GCE node
I20260925 13:12:35.621134 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:35.621394 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:35.622553 5069 hybrid_clock.cc:648] HybridClock initialized: now 1790341955622534 us; error 33 us; skew 500 ppm
I20260925 13:12:35.623883 5069 webserver.cc:460] Webserver started at http://127.0.98.1:43385/ using document root <none> and password file <none>
I20260925 13:12:35.624130 5069 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:35.624174 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:35.624317 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:35.625456 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4b70deaffb849148cc2ba9880329367"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "f0c778437165114d3ea8074965e771a3"
server_key_iv: "323f7cb686d6520d9e2c65107ccb657e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.625886 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4b70deaffb849148cc2ba9880329367"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "f0c778437165114d3ea8074965e771a3"
server_key_iv: "323f7cb686d6520d9e2c65107ccb657e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.627218 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:35.627991 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.628172 5069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:35.628254 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "d4b70deaffb849148cc2ba9880329367"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "f0c778437165114d3ea8074965e771a3"
server_key_iv: "323f7cb686d6520d9e2c65107ccb657e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.628321 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:35.631650 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:35.631930 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:35.632062 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:35.632297 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:35.632628 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:35.632660 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.632680 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:35.632720 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.638902 5069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35325
I20260925 13:12:35.638962 5196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35325 every 8 connection(s)
I20260925 13:12:35.639272 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:35.642536 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.639988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59377 (local address 127.0.98.62:41667)
0925 13:12:35.640053 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.640055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.640330 (+ 275us) server_negotiation.cc:407] Connection header received
0925 13:12:35.640870 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.640872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.640880 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.640931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.642240 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.642242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.642242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.642247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.642262 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.642267 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.642347 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.642442 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.642459 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:12:35.642726 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.639986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41667 (local address 127.0.98.1:59377)
0925 13:12:35.640184 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.640217 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.640746 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.641018 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.641021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.641029 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.642073 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.642073 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.642104 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.642133 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.642347 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.642353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.642356 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.642454 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.642516 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260925 13:12:35.643232 5197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41667
I20260925 13:12:35.643308 5197 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:35.643461 5197 heartbeater.cc:511] Master 127.0.98.62:41667 requested a full tablet report, sending...
I20260925 13:12:35.643980 5009 ts_manager.cc:194] Registered new tserver with Master: d4b70deaffb849148cc2ba9880329367 (127.0.98.1:35325)
I20260925 13:12:35.646042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5069
I20260925 13:12:35.646159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:35.646339 392 external_mini_cluster.cc:1428] Setting key daed52695b4f3b6714822d634fcd5b89
I20260925 13:12:35.647161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--builtin_ntp_servers=127.0.98.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:12:35.729863 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:35.730127 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:35.730166 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:35.731463 5200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:35.731529 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:35.731571 5200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:35.732940 5200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:35.733204 5200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:35.733501 5200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:35.738391 5206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.738406 5200 server_base.cc:1034] running on GCE node
W20260925 13:12:35.738409 5205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.738390 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.738962 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:35.739229 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:35.740394 5200 hybrid_clock.cc:648] HybridClock initialized: now 1790341955740377 us; error 30 us; skew 500 ppm
I20260925 13:12:35.741633 5200 webserver.cc:460] Webserver started at http://127.0.98.2:41107/ using document root <none> and password file <none>
I20260925 13:12:35.741896 5200 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:35.741966 5200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:35.742108 5200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:35.743182 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "4f12cab4efcd9e4987525a000a30f437"
server_key_iv: "ea225b190653efebd7494ffd26a9c44e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.743548 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "4f12cab4efcd9e4987525a000a30f437"
server_key_iv: "ea225b190653efebd7494ffd26a9c44e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.744882 5200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:35.745596 5214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.745791 5200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:35.745889 5200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "4f12cab4efcd9e4987525a000a30f437"
server_key_iv: "ea225b190653efebd7494ffd26a9c44e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.745972 5200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:35.749284 5200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:35.749539 5200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:35.749701 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:35.749924 5200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:35.750231 5200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:35.750283 5200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.750320 5200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:35.750348 5200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.755887 5200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39303
I20260925 13:12:35.755920 5327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39303 every 8 connection(s)
I20260925 13:12:35.756278 5200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:12:35.759265 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.756993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36635 (local address 127.0.98.62:41667)
0925 13:12:35.757081 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.757083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.757318 (+ 235us) server_negotiation.cc:407] Connection header received
0925 13:12:35.757841 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.757842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.757847 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.757909 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.759089 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.759090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.759090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.759092 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.759103 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.759106 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.759162 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.759206 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.759218 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 13:12:35.759450 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.756979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41667 (local address 127.0.98.2:36635)
0925 13:12:35.757192 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.757231 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.757743 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.757981 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.757984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.757992 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.758949 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.758949 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.758976 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.759014 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.759161 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.759163 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.759166 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.759214 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.759257 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260925 13:12:35.759895 5328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41667
I20260925 13:12:35.759981 5328 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:35.760115 5328 heartbeater.cc:511] Master 127.0.98.62:41667 requested a full tablet report, sending...
I20260925 13:12:35.760463 5009 ts_manager.cc:194] Registered new tserver with Master: 0e729504f1bf4f3c9e1afe84c1f11ab3 (127.0.98.2:39303)
I20260925 13:12:35.761854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5200
I20260925 13:12:35.761968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:35.762151 392 external_mini_cluster.cc:1428] Setting key 6538e09ec5e7b463ad78702a201ade1d
I20260925 13:12:35.763031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--builtin_ntp_servers=127.0.98.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:12:35.844277 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:35.844540 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:35.844578 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:35.845878 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:35.845949 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:35.845991 5331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:35.847398 5331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:35.847702 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:35.848009 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:35.853145 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.853170 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:35.853163 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:35.853332 5331 server_base.cc:1034] running on GCE node
I20260925 13:12:35.853570 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:35.853821 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:35.854972 5331 hybrid_clock.cc:648] HybridClock initialized: now 1790341955854967 us; error 34 us; skew 500 ppm
I20260925 13:12:35.856235 5331 webserver.cc:460] Webserver started at http://127.0.98.3:35771/ using document root <none> and password file <none>
I20260925 13:12:35.856475 5331 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:35.856550 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:35.856690 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:35.857860 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "4f45cbf0ea8e4685aa967262caaec0e9"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "a6d04145b81d8482f5eb5f3d40af0338"
server_key_iv: "0698a226aaca0d5b0758928866f05d1e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.858229 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4f45cbf0ea8e4685aa967262caaec0e9"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "a6d04145b81d8482f5eb5f3d40af0338"
server_key_iv: "0698a226aaca0d5b0758928866f05d1e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.859509 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:35.860224 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.860378 5331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:35.860476 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "4f45cbf0ea8e4685aa967262caaec0e9"
format_stamp: "Formatted at 2026-09-25 13:12:35 on dist-test-slave-2d0s"
server_key: "a6d04145b81d8482f5eb5f3d40af0338"
server_key_iv: "0698a226aaca0d5b0758928866f05d1e"
server_key_version: "encryptionkey@0"
I20260925 13:12:35.860548 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:35.863929 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:35.864192 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:35.864323 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:35.864513 5331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:35.864815 5331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:35.864871 5331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.864910 5331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:35.864936 5331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:35.870622 5331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38501
I20260925 13:12:35.870693 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38501 every 8 connection(s)
I20260925 13:12:35.871019 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:12:35.874017 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.871780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49055 (local address 127.0.98.62:41667)
0925 13:12:35.871823 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.871825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.872048 (+ 223us) server_negotiation.cc:407] Connection header received
0925 13:12:35.872567 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.872569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.872575 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.872632 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.873769 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.873771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.873771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.873775 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.873793 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.873798 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.873855 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.873931 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.873946 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:35.874220 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.871748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41667 (local address 127.0.98.3:49055)
0925 13:12:35.871915 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.871948 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.872466 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.872701 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.872705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.872715 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.873578 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.873578 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.873605 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.873631 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.873863 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.873868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.873871 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.873942 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.874008 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 13:12:35.874733 5459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41667
I20260925 13:12:35.874825 5459 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:35.874935 5459 heartbeater.cc:511] Master 127.0.98.62:41667 requested a full tablet report, sending...
I20260925 13:12:35.875245 5009 ts_manager.cc:194] Registered new tserver with Master: 4f45cbf0ea8e4685aa967262caaec0e9 (127.0.98.3:38501)
I20260925 13:12:35.877569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5331
I20260925 13:12:35.877708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:35.877905 392 external_mini_cluster.cc:1428] Setting key 8cfa6b6f9237aea8dfc175176a852912
I20260925 13:12:35.878968 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:12:35.881482 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.880374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.0.98.62:41667)
0925 13:12:35.880416 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.880418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.880601 (+ 183us) server_negotiation.cc:407] Connection header received
0925 13:12:35.880628 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.880629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.880634 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.880693 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.881308 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.881309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.881309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.881313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.881328 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.881332 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.881376 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.881420 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.881432 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:35.883044 5009 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:35.884613 5009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:35.888808 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.886913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.0.1:53608)
0925 13:12:35.887277 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.887323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.887387 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.887680 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.887685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.887692 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.888290 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.888291 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.888309 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.888334 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.888623 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.888630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.888633 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.888709 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.888726 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":70,"threads_started":1}
I20260925 13:12:35.888921 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.886913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38501 (local address 127.0.0.1:54158)
0925 13:12:35.887286 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.887325 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.887385 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.888091 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.888092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.888095 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.888489 (+ 394us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.888490 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.888495 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.888506 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.888786 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.888787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.888787 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.888830 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.888842 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260925 13:12:35.888934 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.887038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.0.98.1:35325)
0925 13:12:35.887533 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.887538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.887551 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:12:35.887580 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.887584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.887594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.887680 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.888434 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.888443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.888443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.888448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.888502 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.888530 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.888595 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.888729 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.888748 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":121,"threads_started":1}
I20260925 13:12:35.889046 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.887154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39303 (local address 127.0.0.1:49890)
0925 13:12:35.887492 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.887503 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.887539 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.888344 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.888345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.888347 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.888665 (+ 318us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.888666 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.888672 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.888683 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.888916 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.888917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.888918 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.888978 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.888991 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":80,"threads_started":1}
I20260925 13:12:35.889218 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.887311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.0.98.3:38501)
0925 13:12:35.887519 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.887524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.887535 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:35.887561 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.887564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.887644 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.887719 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.888628 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.888636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.888637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.888641 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.888674 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.888701 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.888760 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.889111 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.889130 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260925 13:12:35.889281 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.887258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.0.98.2:39303)
0925 13:12:35.888163 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.888167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.888177 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:35.888220 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.888223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.888233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.888325 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.888767 (+ 442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.888771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.888772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.888776 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.888810 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.888830 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.888888 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.889178 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.889194 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":110,"threads_started":1}
I20260925 13:12:35.890008 5393 tablet_service.cc:1467] Processing CreateTablet for tablet a9bde7a7f05e439fac7174c317869e29 (DEFAULT_TABLE table=test-table [id=aa80d91609ae4c79bb34dfc5e69b2f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:35.890025 5262 tablet_service.cc:1467] Processing CreateTablet for tablet a9bde7a7f05e439fac7174c317869e29 (DEFAULT_TABLE table=test-table [id=aa80d91609ae4c79bb34dfc5e69b2f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:35.890026 5131 tablet_service.cc:1467] Processing CreateTablet for tablet a9bde7a7f05e439fac7174c317869e29 (DEFAULT_TABLE table=test-table [id=aa80d91609ae4c79bb34dfc5e69b2f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:35.890369 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bde7a7f05e439fac7174c317869e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.890370 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bde7a7f05e439fac7174c317869e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.890370 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bde7a7f05e439fac7174c317869e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:35.893886 5474 tablet_bootstrap.cc:492] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: Bootstrap starting.
I20260925 13:12:35.894018 5473 tablet_bootstrap.cc:492] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Bootstrap starting.
I20260925 13:12:35.894610 5474 tablet_bootstrap.cc:654] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.894694 5473 tablet_bootstrap.cc:654] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.895002 5474 log.cc:824] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.895040 5473 log.cc:824] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.895380 5475 tablet_bootstrap.cc:492] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Bootstrap starting.
I20260925 13:12:35.895866 5474 tablet_bootstrap.cc:492] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: No bootstrap required, opened a new log
I20260925 13:12:35.895885 5473 tablet_bootstrap.cc:492] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: No bootstrap required, opened a new log
I20260925 13:12:35.895946 5474 ts_tablet_manager.cc:1397] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:35.895944 5475 tablet_bootstrap.cc:654] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:35.895946 5473 ts_tablet_manager.cc:1397] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:35.896215 5475 log.cc:824] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:35.896865 5475 tablet_bootstrap.cc:492] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: No bootstrap required, opened a new log
I20260925 13:12:35.896944 5475 ts_tablet_manager.cc:1397] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:35.898756 5474 raft_consensus.cc:348] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.898945 5474 raft_consensus.cc:374] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.898985 5474 raft_consensus.cc:729] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b70deaffb849148cc2ba9880329367, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.898911 5473 raft_consensus.cc:348] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.899061 5473 raft_consensus.cc:374] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.899106 5473 raft_consensus.cc:729] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f45cbf0ea8e4685aa967262caaec0e9, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.899041 5475 raft_consensus.cc:348] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.899139 5474 consensus_queue.cc:260] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.899215 5475 raft_consensus.cc:374] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:35.899257 5475 raft_consensus.cc:729] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e729504f1bf4f3c9e1afe84c1f11ab3, State: Initialized, Role: FOLLOWER
I20260925 13:12:35.899235 5473 consensus_queue.cc:260] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.899371 5475 consensus_queue.cc:260] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.899456 5474 ts_tablet_manager.cc:1428] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 13:12:35.899492 5473 ts_tablet_manager.cc:1428] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 13:12:35.899603 5197 heartbeater.cc:503] Master 127.0.98.62:41667 was elected leader, sending a full tablet report...
I20260925 13:12:35.899603 5459 heartbeater.cc:503] Master 127.0.98.62:41667 was elected leader, sending a full tablet report...
I20260925 13:12:35.899631 5475 ts_tablet_manager.cc:1428] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:35.899758 5328 heartbeater.cc:503] Master 127.0.98.62:41667 was elected leader, sending a full tablet report...
I20260925 13:12:35.963550 5480 raft_consensus.cc:482] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:35.963754 5480 raft_consensus.cc:504] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.964161 5480 leader_election.cc:288] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e729504f1bf4f3c9e1afe84c1f11ab3 (127.0.98.2:39303), d4b70deaffb849148cc2ba9880329367 (127.0.98.1:35325)
I20260925 13:12:35.965621 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.964453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43735 (local address 127.0.98.2:39303)
0925 13:12:35.964543 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.964546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.964822 (+ 276us) server_negotiation.cc:407] Connection header received
0925 13:12:35.964890 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.964891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.964899 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.964958 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.965404 (+ 446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.965406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.965406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.965411 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.965428 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.965433 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.965484 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.965527 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.965542 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:12:35.965637 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.964453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49097 (local address 127.0.98.1:35325)
0925 13:12:35.964570 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.964572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.964583 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:35.964613 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.964614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.964620 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.964678 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.965374 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.965375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.965375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.965379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.965395 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.965399 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.965481 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.965549 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.965562 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 13:12:35.965642 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.964364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.98.3:49097)
0925 13:12:35.964416 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.964427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.964464 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.964712 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.964713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.964716 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.965233 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.965234 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.965258 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.965274 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.965479 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.965480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.965480 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.965534 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.965549 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:12:35.965708 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.964364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39303 (local address 127.0.98.3:43735)
0925 13:12:35.964728 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:35.964740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:35.964810 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:35.964973 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:35.964974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:35.964976 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:35.965325 (+ 349us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:12:35.965326 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:35.965331 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:12:35.965342 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:35.965479 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:35.965480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:35.965480 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 13:12:35.965528 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:12:35.965542 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":154,"thread_start_us":133,"threads_started":1}
I20260925 13:12:35.966018 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bde7a7f05e439fac7174c317869e29" candidate_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b70deaffb849148cc2ba9880329367" is_pre_election: true
I20260925 13:12:35.966091 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bde7a7f05e439fac7174c317869e29" candidate_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" is_pre_election: true
I20260925 13:12:35.966193 5151 raft_consensus.cc:2454] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f45cbf0ea8e4685aa967262caaec0e9 in term 0.
I20260925 13:12:35.966208 5282 raft_consensus.cc:2454] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f45cbf0ea8e4685aa967262caaec0e9 in term 0.
I20260925 13:12:35.966434 5346 leader_election.cc:302] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f45cbf0ea8e4685aa967262caaec0e9, d4b70deaffb849148cc2ba9880329367; no voters:
I20260925 13:12:35.966557 5480 raft_consensus.cc:2781] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:35.966646 5480 raft_consensus.cc:482] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:35.966682 5480 raft_consensus.cc:3037] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.967438 5480 raft_consensus.cc:504] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.967592 5480 leader_election.cc:288] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0e729504f1bf4f3c9e1afe84c1f11ab3 (127.0.98.2:39303), d4b70deaffb849148cc2ba9880329367 (127.0.98.1:35325)
I20260925 13:12:35.967805 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bde7a7f05e439fac7174c317869e29" candidate_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3"
I20260925 13:12:35.967844 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bde7a7f05e439fac7174c317869e29" candidate_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b70deaffb849148cc2ba9880329367"
I20260925 13:12:35.967893 5282 raft_consensus.cc:3037] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.967909 5151 raft_consensus.cc:3037] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:35.968828 5282 raft_consensus.cc:2454] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f45cbf0ea8e4685aa967262caaec0e9 in term 1.
I20260925 13:12:35.968828 5151 raft_consensus.cc:2454] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f45cbf0ea8e4685aa967262caaec0e9 in term 1.
I20260925 13:12:35.969002 5346 leader_election.cc:302] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f45cbf0ea8e4685aa967262caaec0e9, d4b70deaffb849148cc2ba9880329367; no voters:
I20260925 13:12:35.969105 5480 raft_consensus.cc:2781] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:35.969255 5480 raft_consensus.cc:686] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4f45cbf0ea8e4685aa967262caaec0e9, State: Running, Role: LEADER
I20260925 13:12:35.969355 5480 consensus_queue.cc:237] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } }
I20260925 13:12:35.970281 5007 catalog_manager.cc:5495] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f45cbf0ea8e4685aa967262caaec0e9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f45cbf0ea8e4685aa967262caaec0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:35.982688 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:35.981727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.0.98.3:38501)
0925 13:12:35.981800 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:35.981802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:35.981810 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:35.981843 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:35.981843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:35.981856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:35.981906 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:35.982521 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:35.982522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:35.982523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:35.982526 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:35.982538 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:35.982542 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:35.982582 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:35.982623 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 13:12:35.982634 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 13:12:35.984663 5282 raft_consensus.cc:1261] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 1 FOLLOWER]: Refusing update from remote peer 4f45cbf0ea8e4685aa967262caaec0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:35.984656 5151 raft_consensus.cc:1261] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 1 FOLLOWER]: Refusing update from remote peer 4f45cbf0ea8e4685aa967262caaec0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:35.985021 5480 consensus_queue.cc:1035] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e729504f1bf4f3c9e1afe84c1f11ab3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:35.985189 5483 consensus_queue.cc:1035] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b70deaffb849148cc2ba9880329367" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:35.988870 5488 mvcc.cc:205] Tried to move back new op lower bound from 7333240651710279680 to 7333240651651219456. Current Snapshot: MvccSnapshot[applied={T|T < 7333240651710279680 or (T in {7333240651710279680})}]
I20260925 13:12:35.990623 5007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:35.990707 5007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:35.991422 5007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6910ac093f74037977903ca816fb60f: Sending DeleteTablet for 3 replicas of tablet a9bde7a7f05e439fac7174c317869e29
I20260925 13:12:35.991753 5131 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bde7a7f05e439fac7174c317869e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:35 UTC) from {username='slave'} at 127.0.0.1:53608
I20260925 13:12:35.991756 5393 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bde7a7f05e439fac7174c317869e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:35 UTC) from {username='slave'} at 127.0.0.1:54158
I20260925 13:12:35.991832 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bde7a7f05e439fac7174c317869e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:35 UTC) from {username='slave'} at 127.0.0.1:49890
I20260925 13:12:35.991961 5495 tablet_replica.cc:330] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: stopping tablet replica
I20260925 13:12:35.992056 5495 raft_consensus.cc:2229] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:35.992077 5497 tablet_replica.cc:330] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: stopping tablet replica
I20260925 13:12:35.992161 5495 raft_consensus.cc:2258] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:35.992164 5497 raft_consensus.cc:2229] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:35.992247 5497 raft_consensus.cc:2258] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:35.992314 5496 tablet_replica.cc:330] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: stopping tablet replica
I20260925 13:12:35.992390 5496 raft_consensus.cc:2229] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:35.992501 5496 raft_consensus.cc:2258] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:35.992715 5497 ts_tablet_manager.cc:1905] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:35.992730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5069
I20260925 13:12:35.993000 5495 ts_tablet_manager.cc:1905] T a9bde7a7f05e439fac7174c317869e29 P d4b70deaffb849148cc2ba9880329367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:35.993875 5496 ts_tablet_manager.cc:1905] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:35.995421 5497 ts_tablet_manager.cc:1918] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:35.995487 5497 log.cc:1196] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/a9bde7a7f05e439fac7174c317869e29
I20260925 13:12:35.995612 5496 ts_tablet_manager.cc:1918] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:35.995672 5496 log.cc:1196] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/a9bde7a7f05e439fac7174c317869e29
I20260925 13:12:35.995757 5497 ts_tablet_manager.cc:1939] T a9bde7a7f05e439fac7174c317869e29 P 0e729504f1bf4f3c9e1afe84c1f11ab3: Deleting consensus metadata
I20260925 13:12:35.995918 5496 ts_tablet_manager.cc:1939] T a9bde7a7f05e439fac7174c317869e29 P 4f45cbf0ea8e4685aa967262caaec0e9: Deleting consensus metadata
I20260925 13:12:35.996176 4994 catalog_manager.cc:4841] TS 0e729504f1bf4f3c9e1afe84c1f11ab3 (127.0.98.2:39303): tablet a9bde7a7f05e439fac7174c317869e29 (table test-table [id=aa80d91609ae4c79bb34dfc5e69b2f58]) successfully deleted
I20260925 13:12:35.996279 4994 catalog_manager.cc:4841] TS 4f45cbf0ea8e4685aa967262caaec0e9 (127.0.98.3:38501): tablet a9bde7a7f05e439fac7174c317869e29 (table test-table [id=aa80d91609ae4c79bb34dfc5e69b2f58]) successfully deleted
W20260925 13:12:36.000077 4993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35325 (error 108)
I20260925 13:12:36.000331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5200
I20260925 13:12:36.001402 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.001263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35325 (local address 127.0.0.1:53618)
0925 13:12:36.001324 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:36.001367 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 13:12:36.001520 4993 catalog_manager.cc:4584] TS d4b70deaffb849148cc2ba9880329367 (127.0.98.1:35325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9bde7a7f05e439fac7174c317869e29: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35325: connect: Connection refused (error 111)
I20260925 13:12:36.005282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5331
I20260925 13:12:36.009197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 4978
2026-09-25T13:12:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (658 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 13:12:36.030161 392 mini_oidc.cc:152] Starting JWKS server
I20260925 13:12:36.031272 392 webserver.cc:460] Webserver started at https://127.0.0.1:41705/ using document root <none> and password file <none>
I20260925 13:12:36.031391 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 13:12:36.031585 392 webserver.cc:460] Webserver started at http://127.0.0.1:34993/ using document root <none> and password file <none>
2026-09-25T13:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:36Z Disabled control of system clock
I20260925 13:12:36.039149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35031
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35031
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/testchainca.cert with env {}
W20260925 13:12:36.118839 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:36.119066 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:36.119089 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:36.120203 5504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:36.120237 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:36.120249 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:36.120261 5504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:36.120273 5504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:12:36.120285 5504 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:12:36.120296 5504 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41705/jwks.json
I20260925 13:12:36.121860 5504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:36.122120 5504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:36.122291 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:36.127310 5510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:36.127300 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:36.127300 5509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:36.127671 5504 server_base.cc:1034] running on GCE node
I20260925 13:12:36.127948 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:36.128295 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:36.129467 5504 hybrid_clock.cc:648] HybridClock initialized: now 1790341956129445 us; error 34 us; skew 500 ppm
I20260925 13:12:36.130748 5504 webserver.cc:460] Webserver started at http://127.0.98.62:38837/ using document root <none> and password file <none>
I20260925 13:12:36.131026 5504 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:36.131104 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:36.131274 5504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:36.132334 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "b582318caf8545a091bdef74151952df"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "2045c13dc2f3b1a54a3b2472044076fe"
server_key_iv: "89809ba9ac493f1baae7c975c068eadc"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.132715 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "b582318caf8545a091bdef74151952df"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "2045c13dc2f3b1a54a3b2472044076fe"
server_key_iv: "89809ba9ac493f1baae7c975c068eadc"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.134063 5504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 13:12:36.134786 5518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:36.134950 5504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:36.135059 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "b582318caf8545a091bdef74151952df"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "2045c13dc2f3b1a54a3b2472044076fe"
server_key_iv: "89809ba9ac493f1baae7c975c068eadc"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.135138 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:36.147266 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:36.147603 5504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:36.147760 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:36.151546 5504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35031
I20260925 13:12:36.151597 5570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35031 every 8 connection(s)
I20260925 13:12:36.151962 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:36.152983 5571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:36.153292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5504
I20260925 13:12:36.153409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:36.153601 392 external_mini_cluster.cc:1428] Setting key 0a6feb17e8d99b8f60110e582e6a5cd4
I20260925 13:12:36.156314 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df: Bootstrap starting.
I20260925 13:12:36.157044 5571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:36.157347 5571 log.cc:824] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:36.158152 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df: No bootstrap required, opened a new log
I20260925 13:12:36.159710 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.154155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.0.98.62:35031)
0925 13:12:36.154382 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.154392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.154428 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:12:36.154967 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.154976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.155079 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.155235 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:36.155411 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.156775 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.157690 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.158135 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.158280 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.159052 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:36.159056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:36.159057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:36.159232 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:36.159266 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:36.159278 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:36.159356 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:36.159434 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 13:12:36.159516 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260925 13:12:36.160418 5571 raft_consensus.cc:348] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b582318caf8545a091bdef74151952df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35031 } }
I20260925 13:12:36.160610 5571 raft_consensus.cc:374] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:36.160660 5571 raft_consensus.cc:729] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b582318caf8545a091bdef74151952df, State: Initialized, Role: FOLLOWER
I20260925 13:12:36.160741 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:12:36.160791 5571 consensus_queue.cc:260] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b582318caf8545a091bdef74151952df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35031 } }
I20260925 13:12:36.160859 5571 raft_consensus.cc:388] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:36.160887 5571 raft_consensus.cc:482] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:36.160923 5571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:36.161679 5571 raft_consensus.cc:504] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b582318caf8545a091bdef74151952df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35031 } }
I20260925 13:12:36.161811 5571 leader_election.cc:302] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b582318caf8545a091bdef74151952df; no voters:
I20260925 13:12:36.162029 5571 leader_election.cc:288] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:36.162096 5576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:36.162220 5576 raft_consensus.cc:686] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [term 1 LEADER]: Becoming Leader. State: Replica: b582318caf8545a091bdef74151952df, State: Running, Role: LEADER
I20260925 13:12:36.162324 5571 sys_catalog.cc:564] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:36.162307 5576 consensus_queue.cc:237] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b582318caf8545a091bdef74151952df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35031 } }
I20260925 13:12:36.162706 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b582318caf8545a091bdef74151952df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b582318caf8545a091bdef74151952df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35031 } } }
I20260925 13:12:36.162721 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [sys.catalog]: SysCatalogTable state changed. Reason: New leader b582318caf8545a091bdef74151952df. Latest consensus state: current_term: 1 leader_uuid: "b582318caf8545a091bdef74151952df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b582318caf8545a091bdef74151952df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35031 } } }
I20260925 13:12:36.162807 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.162838 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df [sys.catalog]: This master's current role is: LEADER
W20260925 13:12:36.164024 5592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:12:36.164086 5592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:36.164132 5583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:36.164292 5583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:36.165772 5583 catalog_manager.cc:1347] Generated new cluster ID: 4e5ea969859a409b94aabaa0ab9a4a61
I20260925 13:12:36.165827 5583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:36.195022 5583 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:36.195775 5583 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:36.204093 5583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b582318caf8545a091bdef74151952df: Generated new TSK 0
I20260925 13:12:36.204353 5583 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 13:12:36.236071 5601 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 13:12:36.236183 5601 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 13:12:36.236481 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.221110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.0.98.62:35031)
0925 13:12:36.221161 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.221163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.221282 (+ 119us) server_negotiation.cc:407] Connection header received
0925 13:12:36.221349 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.221350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.221362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.221449 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:12:36.221579 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.222398 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.223194 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.223511 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.223583 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.223762 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:12:36.236213 (+ 12451us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 13:12:36.236382 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 13:12:36.236547 5574 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 13:12:36.236864 5600 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:36.221027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35031 (local address 127.0.0.1:53098)
0925 13:12:36.221204 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:36.221214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:36.221270 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:36.221450 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:36.221450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:36.221452 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:12:36.221505 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.221507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.222493 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.222494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.223112 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.223114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.223585 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.223589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.223632 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.223649 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:12:36.236386 (+ 12737us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 13:12:36.236510 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 13:12:36.237376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5504
2026-09-25T13:12:36Z chronyd exiting
I20260925 13:12:36.273422 392 test_util.cc:175] -----------------------------------------------
I20260925 13:12:36.273506 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790341932215442-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 13:12:36.274403 392 mini_oidc.cc:152] Starting JWKS server
I20260925 13:12:36.275121 392 webserver.cc:460] Webserver started at https://127.0.0.1:36367/ using document root <none> and password file <none>
I20260925 13:12:36.275187 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 13:12:36.275343 392 webserver.cc:460] Webserver started at http://127.0.0.1:36151/ using document root <none> and password file <none>
2026-09-25T13:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:36Z Disabled control of system clock
I20260925 13:12:36.283406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40029
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/testchainca.cert with env {}
W20260925 13:12:36.364132 5607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:36.364368 5607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:36.364391 5607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:36.365882 5607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:36.365957 5607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:36.365985 5607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:36.366004 5607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:36.366026 5607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:12:36.366055 5607 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:12:36.366089 5607 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36367/jwks.json
I20260925 13:12:36.368150 5607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:36.368484 5607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:36.368748 5607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:36.373518 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:36.373600 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:36.373823 5607 server_base.cc:1034] running on GCE node
W20260925 13:12:36.373893 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:36.374187 5607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:36.374516 5607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:36.375667 5607 hybrid_clock.cc:648] HybridClock initialized: now 1790341956375637 us; error 51 us; skew 500 ppm
I20260925 13:12:36.377149 5607 webserver.cc:460] Webserver started at http://127.0.98.62:41095/ using document root <none> and password file <none>
I20260925 13:12:36.377431 5607 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:36.377489 5607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:36.377588 5607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:36.378823 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "12ff6fe2e27f498ca7980525379f84c9"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "65dffe78e9668e9d71fc62d82da7055d"
server_key_iv: "0217711e7a7e2ff185d7cad13882de20"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.379149 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "12ff6fe2e27f498ca7980525379f84c9"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "65dffe78e9668e9d71fc62d82da7055d"
server_key_iv: "0217711e7a7e2ff185d7cad13882de20"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.380498 5607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:36.381227 5621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:36.381421 5607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:36.381542 5607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "12ff6fe2e27f498ca7980525379f84c9"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "65dffe78e9668e9d71fc62d82da7055d"
server_key_iv: "0217711e7a7e2ff185d7cad13882de20"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.381616 5607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:36.390966 5607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:36.391350 5607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:36.391515 5607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:36.395292 5607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40029
I20260925 13:12:36.395352 5673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40029 every 8 connection(s)
I20260925 13:12:36.395691 5607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:36.396751 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:36.397735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5607
I20260925 13:12:36.397840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:36.398064 392 external_mini_cluster.cc:1428] Setting key 4ff5d452c34ca4b75bd648f2078d2f77
I20260925 13:12:36.400137 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9: Bootstrap starting.
I20260925 13:12:36.400882 5674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:36.401196 5674 log.cc:824] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:36.401957 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9: No bootstrap required, opened a new log
I20260925 13:12:36.403658 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.398600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.0.98.62:40029)
0925 13:12:36.398800 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.398807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.398833 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:12:36.399262 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.399269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.399332 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.399453 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:36.399690 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.400641 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.401727 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.402196 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.402291 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.402985 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:36.402989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:36.402989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:36.403159 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:36.403191 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:36.403201 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:36.403251 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:36.403357 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 13:12:36.403421 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260925 13:12:36.404256 5674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ff6fe2e27f498ca7980525379f84c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260925 13:12:36.404482 5674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:36.404539 5674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ff6fe2e27f498ca7980525379f84c9, State: Initialized, Role: FOLLOWER
I20260925 13:12:36.404693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:12:36.404673 5674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ff6fe2e27f498ca7980525379f84c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260925 13:12:36.404749 5674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:36.404795 5674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:36.404845 5674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:36.405552 5674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ff6fe2e27f498ca7980525379f84c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260925 13:12:36.405670 5674 leader_election.cc:302] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ff6fe2e27f498ca7980525379f84c9; no voters:
I20260925 13:12:36.405884 5674 leader_election.cc:288] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:36.405936 5679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:36.406095 5679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [term 1 LEADER]: Becoming Leader. State: Replica: 12ff6fe2e27f498ca7980525379f84c9, State: Running, Role: LEADER
I20260925 13:12:36.406126 5674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:36.406188 5679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ff6fe2e27f498ca7980525379f84c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260925 13:12:36.406536 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ff6fe2e27f498ca7980525379f84c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ff6fe2e27f498ca7980525379f84c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } } }
I20260925 13:12:36.406548 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ff6fe2e27f498ca7980525379f84c9. Latest consensus state: current_term: 1 leader_uuid: "12ff6fe2e27f498ca7980525379f84c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ff6fe2e27f498ca7980525379f84c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } } }
I20260925 13:12:36.406658 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.406688 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.407184 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:36.407687 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:36.409191 5688 catalog_manager.cc:1347] Generated new cluster ID: fc0abe655c424b60803f72f07a22e0b5
I20260925 13:12:36.409241 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:36.455569 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:36.456331 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:36.465962 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ff6fe2e27f498ca7980525379f84c9: Generated new TSK 0
I20260925 13:12:36.466192 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 13:12:36.493057 5704 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 13:12:36.493170 5704 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 13:12:36.493523 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.475025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.0.98.62:40029)
0925 13:12:36.475067 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.475069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.475368 (+ 299us) server_negotiation.cc:407] Connection header received
0925 13:12:36.475534 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.475536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.475550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.475596 (+ 46us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:12:36.475824 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.476582 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.477786 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.478076 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.478120 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.478366 (+ 246us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:12:36.493262 (+ 14896us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 13:12:36.493436 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 13:12:36.493588 5677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 13:12:36.493570 5703 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:36.475117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40029 (local address 127.0.0.1:41584)
0925 13:12:36.475301 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:36.475313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:36.475435 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:36.475672 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:36.475673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:36.475675 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:12:36.475743 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.475746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.476717 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.476718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.477654 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.477657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.478187 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.478191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.478248 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.478253 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:12:36.493397 (+ 15144us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 13:12:36.493499 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260925 13:12:36.493989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5607
2026-09-25T13:12:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (257 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 13:12:36.531924 392 mini_oidc.cc:152] Starting JWKS server
I20260925 13:12:36.532593 392 webserver.cc:460] Webserver started at https://127.0.0.1:35587/ using document root <none> and password file <none>
I20260925 13:12:36.532640 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 13:12:36.532779 392 webserver.cc:460] Webserver started at http://127.0.0.1:43307/ using document root <none> and password file <none>
2026-09-25T13:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:36Z Disabled control of system clock
I20260925 13:12:36.540438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35587/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 13:12:36.622160 5710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:36.622433 5710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:36.622474 5710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:36.623703 5710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:36.623769 5710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:36.623801 5710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:36.623827 5710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:36.623858 5710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:12:36.623881 5710 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:12:36.623901 5710 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35587/jwks.json
I20260925 13:12:36.625270 5710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35587/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:36.625558 5710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:36.625869 5710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:36.630743 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:36.630811 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:36.630964 5718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:36.630978 5710 server_base.cc:1034] running on GCE node
I20260925 13:12:36.631278 5710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:36.631633 5710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:36.632802 5710 hybrid_clock.cc:648] HybridClock initialized: now 1790341956632779 us; error 33 us; skew 500 ppm
I20260925 13:12:36.634222 5710 webserver.cc:460] Webserver started at http://127.0.98.62:42057/ using document root <none> and password file <none>
I20260925 13:12:36.634508 5710 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:36.634589 5710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:36.634744 5710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:36.635872 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "27192ec9e21942b480ef458572eaab67"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "79e3c4c886f10005a0cda225ae2a8357"
server_key_iv: "4a44bdf7678c440435c98889c9c7c7df"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.636228 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "27192ec9e21942b480ef458572eaab67"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "79e3c4c886f10005a0cda225ae2a8357"
server_key_iv: "4a44bdf7678c440435c98889c9c7c7df"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.637540 5710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:36.638260 5724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:36.638413 5710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:36.638535 5710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "27192ec9e21942b480ef458572eaab67"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "79e3c4c886f10005a0cda225ae2a8357"
server_key_iv: "4a44bdf7678c440435c98889c9c7c7df"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.638604 5710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:36.656035 5710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:36.656376 5710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:36.656553 5710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:36.660799 5710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36601
I20260925 13:12:36.660835 5776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36601 every 8 connection(s)
I20260925 13:12:36.661177 5710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:36.661872 5777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:36.664397 5777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67: Bootstrap starting.
I20260925 13:12:36.664981 5777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:36.665180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5710
I20260925 13:12:36.665279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:36.665333 5777 log.cc:824] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:36.665462 392 external_mini_cluster.cc:1428] Setting key 53c9eee2acdb2a2f8ae7880f8400a97d
I20260925 13:12:36.666236 5777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67: No bootstrap required, opened a new log
I20260925 13:12:36.669030 5777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27192ec9e21942b480ef458572eaab67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36601 } }
I20260925 13:12:36.669195 5777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:36.669219 5777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27192ec9e21942b480ef458572eaab67, State: Initialized, Role: FOLLOWER
I20260925 13:12:36.669389 5777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27192ec9e21942b480ef458572eaab67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36601 } }
I20260925 13:12:36.669459 5777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:36.669513 5777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:36.669559 5777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:36.670363 5777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27192ec9e21942b480ef458572eaab67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36601 } }
I20260925 13:12:36.670475 5777 leader_election.cc:302] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27192ec9e21942b480ef458572eaab67; no voters:
I20260925 13:12:36.670656 5777 leader_election.cc:288] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:36.670830 5783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:36.670938 5777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:36.671046 5783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [term 1 LEADER]: Becoming Leader. State: Replica: 27192ec9e21942b480ef458572eaab67, State: Running, Role: LEADER
I20260925 13:12:36.671145 5783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27192ec9e21942b480ef458572eaab67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36601 } }
I20260925 13:12:36.671552 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27192ec9e21942b480ef458572eaab67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27192ec9e21942b480ef458572eaab67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36601 } } }
I20260925 13:12:36.671640 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.671792 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27192ec9e21942b480ef458572eaab67. Latest consensus state: current_term: 1 leader_uuid: "27192ec9e21942b480ef458572eaab67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27192ec9e21942b480ef458572eaab67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36601 } } }
I20260925 13:12:36.671861 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.671980 5791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:36.672490 5791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:36.672564 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.665985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.98.62:36601)
0925 13:12:36.666202 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.666211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.666242 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:12:36.666808 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.666817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.666896 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.667058 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:36.667250 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.668708 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.669909 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.670583 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.670746 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.671529 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:36.671534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:36.671535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:36.671781 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:36.671825 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:36.671840 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:36.671903 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:36.672202 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 13:12:36.672294 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260925 13:12:36.673588 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:12:36.674134 5791 catalog_manager.cc:1347] Generated new cluster ID: 1edf419162eb40b6bd98a63fea40ffa9
I20260925 13:12:36.674216 5791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:36.687079 5791 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:36.687685 5791 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:36.696561 5791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27192ec9e21942b480ef458572eaab67: Generated new TSK 0
I20260925 13:12:36.696805 5791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:12:36.714649 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.709880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.0.98.62:36601)
0925 13:12:36.709949 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.709951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.710057 (+ 106us) server_negotiation.cc:407] Connection header received
0925 13:12:36.710144 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.710145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.710159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.710236 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:12:36.710418 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.711247 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.712078 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.712403 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.712508 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.712758 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:12:36.714421 (+ 1663us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35587/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 13:12:36.714573 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57676: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35587/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 13:12:36.714720 5781 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57676: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35587/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 13:12:36.716493 5806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:36.709787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36601 (local address 127.0.0.1:57676)
0925 13:12:36.709972 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:36.709985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:36.710057 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:36.710272 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:36.710273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:36.710276 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:12:36.710334 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.710336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.711347 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.711348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.711991 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.711993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.712556 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.712564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.712628 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.712635 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:12:36.714528 (+ 1893us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35587/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 13:12:36.716396 (+ 1868us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35587/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1674,"thread_start_us":115,"threads_started":1}
I20260925 13:12:36.716881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5710
E20260925 13:12:36.725556 5807 webserver.cc:533] Webserver: sslize failed:
E20260925 13:12:36.725644 5807 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T13:12:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T13:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:36Z Disabled control of system clock
I20260925 13:12:36.762007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46843
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46843
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/testchainca.cert with env {}
W20260925 13:12:36.843144 5811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:36.843353 5811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:36.843376 5811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:36.844518 5811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:36.844550 5811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:36.844563 5811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:36.844575 5811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:36.844587 5811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:12:36.844599 5811 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:12:36.844610 5811 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 13:12:36.846423 5811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:36.846680 5811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:36.846887 5811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:36.851836 5819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:36.851926 5811 server_base.cc:1034] running on GCE node
W20260925 13:12:36.851859 5817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:36.851840 5816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:36.852291 5811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:36.852638 5811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:36.853747 5811 hybrid_clock.cc:648] HybridClock initialized: now 1790341956853735 us; error 28 us; skew 500 ppm
I20260925 13:12:36.855091 5811 webserver.cc:460] Webserver started at http://127.0.98.62:36397/ using document root <none> and password file <none>
I20260925 13:12:36.855352 5811 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:36.855399 5811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:36.855489 5811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:36.856504 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "f2791c00657a4c4bb3e8800145da0875"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "4768ee2d46023a5b61ccc5c941a7f64b"
server_key_iv: "1953e751acf6607722b4f1fea56a0c35"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.856806 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2791c00657a4c4bb3e8800145da0875"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "4768ee2d46023a5b61ccc5c941a7f64b"
server_key_iv: "1953e751acf6607722b4f1fea56a0c35"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.858150 5811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:36.858836 5825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:36.859040 5811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:36.859104 5811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "f2791c00657a4c4bb3e8800145da0875"
format_stamp: "Formatted at 2026-09-25 13:12:36 on dist-test-slave-2d0s"
server_key: "4768ee2d46023a5b61ccc5c941a7f64b"
server_key_iv: "1953e751acf6607722b4f1fea56a0c35"
server_key_version: "encryptionkey@0"
I20260925 13:12:36.859192 5811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:36.878780 5811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:36.879132 5811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:36.879304 5811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:36.883405 5811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46843
I20260925 13:12:36.883481 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46843 every 8 connection(s)
I20260925 13:12:36.883836 5811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:36.884770 5878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:36.886245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5811
I20260925 13:12:36.886336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:36.886557 392 external_mini_cluster.cc:1428] Setting key 6d42c4076c2810714be6efe36b8ddc61
I20260925 13:12:36.887639 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875: Bootstrap starting.
I20260925 13:12:36.888403 5878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:36.888715 5878 log.cc:824] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:36.889454 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875: No bootstrap required, opened a new log
I20260925 13:12:36.891745 5878 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2791c00657a4c4bb3e8800145da0875" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46843 } }
I20260925 13:12:36.891880 5878 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:36.891928 5878 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2791c00657a4c4bb3e8800145da0875, State: Initialized, Role: FOLLOWER
I20260925 13:12:36.892076 5878 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2791c00657a4c4bb3e8800145da0875" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46843 } }
I20260925 13:12:36.892168 5878 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:36.892201 5878 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:36.892263 5878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:36.892701 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.887081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.0.98.62:46843)
0925 13:12:36.887300 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.887311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.887345 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:12:36.887885 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.887895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.887988 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.888167 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:36.888384 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.889775 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.890538 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.891167 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.891330 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.891973 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:36.891977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:36.891978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:12:36.892181 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:36.892227 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:12:36.892243 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:36.892310 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:36.892393 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 13:12:36.892483 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260925 13:12:36.892946 5878 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2791c00657a4c4bb3e8800145da0875" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46843 } }
I20260925 13:12:36.893070 5878 leader_election.cc:302] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2791c00657a4c4bb3e8800145da0875; no voters:
I20260925 13:12:36.893218 5878 leader_election.cc:288] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:36.893234 5883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:36.893338 5883 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [term 1 LEADER]: Becoming Leader. State: Replica: f2791c00657a4c4bb3e8800145da0875, State: Running, Role: LEADER
I20260925 13:12:36.893405 5883 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2791c00657a4c4bb3e8800145da0875" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46843 } }
I20260925 13:12:36.893462 5878 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:36.893570 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:12:36.893740 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2791c00657a4c4bb3e8800145da0875. Latest consensus state: current_term: 1 leader_uuid: "f2791c00657a4c4bb3e8800145da0875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2791c00657a4c4bb3e8800145da0875" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46843 } } }
I20260925 13:12:36.893749 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2791c00657a4c4bb3e8800145da0875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2791c00657a4c4bb3e8800145da0875" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46843 } } }
I20260925 13:12:36.893815 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.893843 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:36.894011 5888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:36.894477 5888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:36.896111 5888 catalog_manager.cc:1347] Generated new cluster ID: 896cf4a3718a43288adb33f2dfec0c12
I20260925 13:12:36.896159 5888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:36.902419 5888 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:36.903116 5888 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:36.909902 5888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2791c00657a4c4bb3e8800145da0875: Generated new TSK 0
I20260925 13:12:36.910140 5888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:12:37.004136 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:36.916050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.0.98.62:46843)
0925 13:12:36.916095 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:36.916097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:36.916194 (+ 97us) server_negotiation.cc:407] Connection header received
0925 13:12:36.916261 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:36.916262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:36.916272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:36.916326 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:12:36.916444 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.916902 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.918074 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.918263 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.918317 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.918516 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:12:37.003919 (+ 85403us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 13:12:37.004078 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:12:37.004230 5881 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 13:12:37.004341 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:36.915953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46843 (local address 127.0.0.1:56450)
0925 13:12:36.916122 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:36.916134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:36.916198 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:36.916334 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:36.916335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:36.916337 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:12:36.916391 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.916393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.916990 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.916991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.917986 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:36.917989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:36.918343 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:36.918349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:36.918408 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:36.918414 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:12:37.004083 (+ 85669us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 13:12:37.004231 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260925 13:12:37.004691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5811
2026-09-25T13:12:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (267 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:37 dist-test-slave-2d0s krb5kdc[5919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:37 dist-test-slave-2d0s krb5kdc[5919](info): set up 2 sockets
Sep 25 13:12:37 dist-test-slave-2d0s krb5kdc[5919](info): commencing operation
krb5kdc: starting...
W20260925 13:12:38.197959 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:38.313994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:38.392820 5932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:38.393057 5932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:38.393095 5932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:38.394343 5932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:38.394404 5932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:38.394428 5932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:38.394448 5932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:38.394477 5932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:38.395959 5932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:38.396222 5932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:38.396449 5932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:38.401211 5940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:38.401242 5938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:38.401414 5932 server_base.cc:1034] running on GCE node
W20260925 13:12:38.401508 5937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:38.401799 5932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:38.402071 5932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:38.403208 5932 hybrid_clock.cc:648] HybridClock initialized: now 1790341958403195 us; error 27 us; skew 500 ppm
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.408661 5932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:38.409056 5932 webserver.cc:460] Webserver started at http://127.0.98.62:44647/ using document root <none> and password file <none>
I20260925 13:12:38.409247 5932 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:38.409287 5932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:38.409371 5932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:38.410440 5932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "8bda26ba82e37b76a7d125c49ce6e57e"
server_key_iv: "b8d8017766985fde59511bba4fc726e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.410734 5932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "8bda26ba82e37b76a7d125c49ce6e57e"
server_key_iv: "b8d8017766985fde59511bba4fc726e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.411965 5932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:38.412612 5947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.412820 5932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:38.412922 5932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "8bda26ba82e37b76a7d125c49ce6e57e"
server_key_iv: "b8d8017766985fde59511bba4fc726e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.412993 5932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:38.438746 5932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:38.441061 5932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:38.441219 5932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:38.445109 5932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36053
I20260925 13:12:38.445183 5999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36053 every 8 connection(s)
I20260925 13:12:38.445639 5932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:38.446274 6000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:38.448469 6000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2: Bootstrap starting.
I20260925 13:12:38.448711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 5932
I20260925 13:12:38.448837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:38.449038 392 external_mini_cluster.cc:1428] Setting key a1f00c90a8c9515c8dfb0feeb6cccf54
I20260925 13:12:38.449239 6000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:38.449579 6000 log.cc:824] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:38.450268 6000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2: No bootstrap required, opened a new log
I20260925 13:12:38.452549 6000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36053 } }
I20260925 13:12:38.452711 6000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:38.452744 6000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7eea1bb68341188dcb0f8ae4df5fd2, State: Initialized, Role: FOLLOWER
I20260925 13:12:38.452852 6000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36053 } }
I20260925 13:12:38.452919 6000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:38.452943 6000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:38.452986 6000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:38.453511 6000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36053 } }
I20260925 13:12:38.453639 6000 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7eea1bb68341188dcb0f8ae4df5fd2; no voters:
I20260925 13:12:38.453830 6000 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:38.453881 6005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:38.454027 6005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7eea1bb68341188dcb0f8ae4df5fd2, State: Running, Role: LEADER
I20260925 13:12:38.454111 6000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:38.454123 6005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36053 } }
I20260925 13:12:38.454456 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7eea1bb68341188dcb0f8ae4df5fd2. Latest consensus state: current_term: 1 leader_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36053 } } }
I20260925 13:12:38.454533 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:38.455148 6011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:38.455591 6011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:38.455739 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7eea1bb68341188dcb0f8ae4df5fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36053 } } }
I20260925 13:12:38.455811 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2 [sys.catalog]: This master's current role is: LEADER
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:38.457062 6011 catalog_manager.cc:1347] Generated new cluster ID: 67574acfa1a54d3384586831037e66d3
I20260925 13:12:38.457110 6011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:38.460747 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.449551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.0.98.62:36053)
0925 13:12:38.449773 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.449783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.449818 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:12:38.450208 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.450214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.450258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.450416 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.451199 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.452063 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.453125 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.453510 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.453617 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.456621 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.456625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.456626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.456636 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.459126 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.459540 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.459543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.459543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.459652 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.459878 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.459879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.459879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.460155 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.460297 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.460422 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.460549 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260925 13:12:38.466318 6011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:38.466981 6011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:38.471719 6011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7eea1bb68341188dcb0f8ae4df5fd2: Generated new TSK 0
I20260925 13:12:38.471880 6011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:38.496405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36053
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:38.576773 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:38.576954 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:38.576977 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:38.578137 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:38.578178 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:38.578204 6026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:38.579636 6026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:38.579878 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:38.580092 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:38.585035 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:38.585058 6026 server_base.cc:1034] running on GCE node
W20260925 13:12:38.585012 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:38.585022 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:38.585409 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:38.585644 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:38.586833 6026 hybrid_clock.cc:648] HybridClock initialized: now 1790341958586827 us; error 38 us; skew 500 ppm
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.589323 6026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:38.589800 6026 webserver.cc:460] Webserver started at http://127.0.98.1:35723/ using document root <none> and password file <none>
I20260925 13:12:38.590005 6026 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:38.590075 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:38.590199 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:38.591144 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a7736c348804ac1be293b77b15f5b6b"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "6cbc8e87a133ff141ce4058f948f24cb"
server_key_iv: "50f2b00ae30bd0b8683115bc6104a6b8"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.591521 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a7736c348804ac1be293b77b15f5b6b"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "6cbc8e87a133ff141ce4058f948f24cb"
server_key_iv: "50f2b00ae30bd0b8683115bc6104a6b8"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.592761 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:38.593389 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.593554 6026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:38.593644 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "3a7736c348804ac1be293b77b15f5b6b"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "6cbc8e87a133ff141ce4058f948f24cb"
server_key_iv: "50f2b00ae30bd0b8683115bc6104a6b8"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.593725 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:38.605051 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:38.607298 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:38.607451 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:38.607663 6026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:38.607937 6026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:38.607988 6026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.608031 6026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:38.608079 6026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.613611 6026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39207
I20260925 13:12:38.613725 6154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39207 every 8 connection(s)
I20260925 13:12:38.614048 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:38.621114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6026
I20260925 13:12:38.621230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:38.621410 392 external_mini_cluster.cc:1428] Setting key 4696a4ad8b19d53e36ce2fa5bea50ee1
I20260925 13:12:38.623144 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.614744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33397 (local address 127.0.98.62:36053)
0925 13:12:38.614802 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.614803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.614974 (+ 171us) server_negotiation.cc:407] Connection header received
0925 13:12:38.615899 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.615899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.615908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.615961 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.616512 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.617117 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.618096 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.618291 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.618343 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.620211 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.620213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.620214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.620227 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.621745 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.622192 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.622193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.622194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.622258 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.622500 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.622501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.622501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.622617 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.622688 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.622959 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.623057 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:12:38.623282 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.614708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36053 (local address 127.0.98.1:33397)
0925 13:12:38.614851 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:38.614888 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:38.615813 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:38.616019 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:38.616022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:38.616266 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:38.616443 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:38.616446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.617191 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.617192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:38.618024 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:38.618026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.618362 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.618363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:38.618452 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.618823 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:38.618830 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:38.620115 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:38.621824 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:38.621829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:38.621831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:38.622099 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:38.622330 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:38.622331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:38.622331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:38.622436 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:38.622688 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:38.622695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:38.622828 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 13:12:38.622924 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 13:12:38.623067 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260925 13:12:38.623792 6155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36053
I20260925 13:12:38.623864 6155 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:38.624046 6155 heartbeater.cc:511] Master 127.0.98.62:36053 requested a full tablet report, sending...
I20260925 13:12:38.624547 5964 ts_manager.cc:194] Registered new tserver with Master: 3a7736c348804ac1be293b77b15f5b6b (127.0.98.1:39207)
I20260925 13:12:38.625113 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33397
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:38.644972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36053
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:38.724193 6160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:38.724403 6160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:38.724449 6160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:38.725483 6160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:38.725543 6160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:38.725580 6160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:38.726963 6160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:38.727214 6160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:38.727430 6160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:38.732277 6166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:38.732275 6168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:38.732275 6165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:38.732537 6160 server_base.cc:1034] running on GCE node
I20260925 13:12:38.732826 6160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:38.733057 6160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:38.734215 6160 hybrid_clock.cc:648] HybridClock initialized: now 1790341958734209 us; error 38 us; skew 500 ppm
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.736528 6160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:38.736948 6160 webserver.cc:460] Webserver started at http://127.0.98.2:37901/ using document root <none> and password file <none>
I20260925 13:12:38.737155 6160 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:38.737195 6160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:38.737275 6160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:38.738281 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "185d5871751549a9943ebdbd637443e9"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "97348880775c44677f2fe45eea13f2d0"
server_key_iv: "29c0c7ab78456822313d95e12c564b4f"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.738560 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "185d5871751549a9943ebdbd637443e9"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "97348880775c44677f2fe45eea13f2d0"
server_key_iv: "29c0c7ab78456822313d95e12c564b4f"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.739773 6160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:38.740464 6175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.740671 6160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:38.740756 6160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "185d5871751549a9943ebdbd637443e9"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "97348880775c44677f2fe45eea13f2d0"
server_key_iv: "29c0c7ab78456822313d95e12c564b4f"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.740811 6160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:38.751125 6160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:38.753490 6160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:38.753615 6160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:38.753922 6160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:38.754225 6160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:38.754261 6160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.754307 6160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:38.754364 6160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.760545 6160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34145
I20260925 13:12:38.760615 6288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34145 every 8 connection(s)
I20260925 13:12:38.760912 6160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:38.769351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6160
I20260925 13:12:38.769454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:38.769460 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.761558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60879 (local address 127.0.98.62:36053)
0925 13:12:38.761600 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.761603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.761798 (+ 195us) server_negotiation.cc:407] Connection header received
0925 13:12:38.762616 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.762617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.762627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.762685 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.763194 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.763675 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.764589 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.764776 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.764837 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.766849 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.766851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.766851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.766857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.768267 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.768604 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.768605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.768605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.768671 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.768875 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.768876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.768876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.768972 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.769021 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.769271 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.769356 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:38.769500 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.761518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36053 (local address 127.0.98.2:60879)
0925 13:12:38.761689 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:38.761727 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:38.762528 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:38.762731 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:38.762734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:38.762941 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:38.763107 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:38.763111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.763748 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.763749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:38.764490 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:38.764492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.764852 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.764853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:38.764944 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.765353 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:38.765361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:38.766724 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:38.768332 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:38.768337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:38.768338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:38.768526 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:38.768736 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:38.768737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:38.768737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:38.768820 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:38.769023 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:38.769026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:38.769123 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 13:12:38.769212 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:12:38.769317 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260925 13:12:38.769656 392 external_mini_cluster.cc:1428] Setting key bd1ea2aa5d766e4d5505ce74c039d8fa
I20260925 13:12:38.770053 6289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36053
I20260925 13:12:38.770125 6289 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:38.770291 6289 heartbeater.cc:511] Master 127.0.98.62:36053 requested a full tablet report, sending...
I20260925 13:12:38.770658 5964 ts_manager.cc:194] Registered new tserver with Master: 185d5871751549a9943ebdbd637443e9 (127.0.98.2:34145)
I20260925 13:12:38.771127 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60879
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:38.792970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36053
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:38.873494 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:38.873746 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:38.873785 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:38.874922 6294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:38.874990 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:38.875029 6294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:38.876435 6294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:38.876699 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:38.876924 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:38.881819 6300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:38.881820 6299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:38.881835 6302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:38.882056 6294 server_base.cc:1034] running on GCE node
I20260925 13:12:38.882256 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:38.882484 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:38.883622 6294 hybrid_clock.cc:648] HybridClock initialized: now 1790341958883627 us; error 42 us; skew 500 ppm
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.886137 6294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:38.886495 6294 webserver.cc:460] Webserver started at http://127.0.98.3:34603/ using document root <none> and password file <none>
I20260925 13:12:38.886684 6294 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:38.886722 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:38.886818 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:38.887844 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "c84dab16e80e475f916ef71f8a4099a0"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "88658ebcfdae23d5e0682ef662dd922d"
server_key_iv: "4b92f431f4ea2780b2ea9008804a1a33"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.888202 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c84dab16e80e475f916ef71f8a4099a0"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "88658ebcfdae23d5e0682ef662dd922d"
server_key_iv: "4b92f431f4ea2780b2ea9008804a1a33"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.889478 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:38.890271 6309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.890440 6294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.890527 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "c84dab16e80e475f916ef71f8a4099a0"
format_stamp: "Formatted at 2026-09-25 13:12:38 on dist-test-slave-2d0s"
server_key: "88658ebcfdae23d5e0682ef662dd922d"
server_key_iv: "4b92f431f4ea2780b2ea9008804a1a33"
server_key_version: "encryptionkey@0"
I20260925 13:12:38.890636 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:38.909852 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:38.912258 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:38.912424 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:38.912734 6294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:38.913036 6294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:38.913089 6294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.913129 6294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:38.913156 6294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:38.919263 6294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40547
I20260925 13:12:38.919368 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40547 every 8 connection(s)
I20260925 13:12:38.919674 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:38.927488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6294
I20260925 13:12:38.927568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:38.927788 392 external_mini_cluster.cc:1428] Setting key a24fa496d78409ffca4204dc48f7b807
I20260925 13:12:38.929117 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.920434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37359 (local address 127.0.98.62:36053)
0925 13:12:38.920479 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.920481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.920714 (+ 233us) server_negotiation.cc:407] Connection header received
0925 13:12:38.921730 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.921731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.921745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.921795 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.922444 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.922989 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.924014 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.924216 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.924288 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.926405 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.926406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.926406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.926412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.927672 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.928113 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.928115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.928116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.928179 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.928425 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.928426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.928427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.928547 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.928626 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.928946 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.929021 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:38.929315 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.920415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36053 (local address 127.0.98.3:37359)
0925 13:12:38.920575 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:38.920621 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:38.921581 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:38.921868 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:38.921872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:38.922163 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:38.922349 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:38.922354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.923098 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.923098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:38.923911 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:38.923912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.924306 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.924307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:38.924432 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.924964 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:38.924972 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:38.926319 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:38.927743 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:38.927750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:38.927752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:38.928030 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:38.928246 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:38.928247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:38.928247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:38.928352 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:38.928617 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:38.928623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:38.928795 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 13:12:38.928925 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 13:12:38.929082 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260925 13:12:38.929781 6423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36053
I20260925 13:12:38.929862 6423 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:38.929993 6423 heartbeater.cc:511] Master 127.0.98.62:36053 requested a full tablet report, sending...
I20260925 13:12:38.930305 5964 ts_manager.cc:194] Registered new tserver with Master: c84dab16e80e475f916ef71f8a4099a0 (127.0.98.3:40547)
I20260925 13:12:38.930681 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37359
I20260925 13:12:38.939396 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/test-admin.keytab.
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.954698 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 13:12:38.956483 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:38.955610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.98.1:39207)
0925 13:12:38.955787 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.955792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.955803 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:38.955828 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.955829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.955843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.955910 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.956395 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51940: BlockingRecv error: recv got EOF from 127.0.0.1:51940 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
W20260925 13:12:38.956573 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.957633 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.958575 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:38.966377 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.959097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.0.98.1:39207)
0925 13:12:38.959151 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.959153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.959182 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:12:38.959246 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.959246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.959256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.959302 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.959778 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.960499 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.961228 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.961540 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.961595 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.963612 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.963619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.963619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.963625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.965229 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.965570 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.965574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.965574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.965642 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.965927 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.965928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.965928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.966078 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.966187 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.966247 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.966303 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 13:12:38.968199 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:38.967629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.0.98.1:39207)
0925 13:12:38.967690 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.967692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.967753 (+ 61us) server_negotiation.cc:407] Connection header received
0925 13:12:38.967833 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.967834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.967844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.967888 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.968154 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51952: BlockingRecv error: recv got EOF from 127.0.0.1:51952 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 13:12:38.968276 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.969167 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.969993 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:38.977001 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.970435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.0.98.1:39207)
0925 13:12:38.970471 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.970473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.970509 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:12:38.970564 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.970564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.970572 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.970619 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.970989 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.971459 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.972297 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.972494 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.972581 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.974673 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.974675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.974675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.974680 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.976011 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.976332 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.976333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.976333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.976379 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.976562 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.976563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.976563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.976645 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.976714 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.976819 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.976871 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 13:12:38.978504 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:38.977967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.98.1:39207)
0925 13:12:38.978027 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.978029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.978089 (+ 60us) server_negotiation.cc:407] Connection header received
0925 13:12:38.978144 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.978145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.978154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.978198 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.978456 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51962: BlockingRecv error: recv got EOF from 127.0.0.1:51962 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 13:12:38.978621 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790341932215442-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.979516 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:38.980351 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 13:12:38 dist-test-slave-2d0s krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:12:38.987008 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:38.980829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.0.98.1:39207)
0925 13:12:38.980882 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:38.980883 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:38.980895 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:12:38.980941 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:38.980942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:38.980948 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:38.980991 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:38.981303 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.981845 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.982474 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:38.982666 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:38.982718 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:38.984693 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:38.984695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:38.984695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:38.984701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:38.986077 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.986383 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.986383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.986384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.986430 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:38.986613 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:38.986613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:38.986614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:38.986696 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:38.986774 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:38.986858 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 13:12:38.986906 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:12:38.987705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6026
I20260925 13:12:38.992889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6160
I20260925 13:12:38.997164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6294
I20260925 13:12:39.001331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 5932
2026-09-25T13:12:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1822 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:39 dist-test-slave-2d0s krb5kdc[6443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:39 dist-test-slave-2d0s krb5kdc[6443](info): set up 2 sockets
Sep 25 13:12:39 dist-test-slave-2d0s krb5kdc[6443](info): commencing operation
krb5kdc: starting...
W20260925 13:12:40.022130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 13:12:40.135990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42953
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:40.213795 6456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:40.214018 6456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:40.214056 6456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:40.215183 6456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:40.215253 6456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:40.215279 6456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:40.215312 6456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:40.215334 6456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:40.216778 6456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:40.217049 6456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:40.217250 6456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:40.221932 6464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.221937 6462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.222028 6456 server_base.cc:1034] running on GCE node
W20260925 13:12:40.221944 6461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.222440 6456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:40.222728 6456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:40.223881 6456 hybrid_clock.cc:648] HybridClock initialized: now 1790341960223868 us; error 26 us; skew 500 ppm
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:40.226284 6456 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 13:12:40.226751 6456 webserver.cc:460] Webserver started at http://127.0.98.62:37213/ using document root <none> and password file <none>
I20260925 13:12:40.226972 6456 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:40.227035 6456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:40.227181 6456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:40.228168 6456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "b382c1aff28e4d7c875ca75c180ace1e"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "06c07ca34d89a712ff1581bd9602f271"
server_key_iv: "a1bbe5b2a339b3900ed9e7c24cac6b00"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.228502 6456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "b382c1aff28e4d7c875ca75c180ace1e"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "06c07ca34d89a712ff1581bd9602f271"
server_key_iv: "a1bbe5b2a339b3900ed9e7c24cac6b00"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.229750 6456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:40.230432 6471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.230599 6456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:40.230706 6456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "b382c1aff28e4d7c875ca75c180ace1e"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "06c07ca34d89a712ff1581bd9602f271"
server_key_iv: "a1bbe5b2a339b3900ed9e7c24cac6b00"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.230786 6456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:40.249392 6456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:40.251598 6456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:40.251752 6456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:40.255604 6456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42953
I20260925 13:12:40.255632 6523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42953 every 8 connection(s)
I20260925 13:12:40.255959 6456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:40.256834 6524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:40.259020 6524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Bootstrap starting.
I20260925 13:12:40.259510 6524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:40.259776 6524 log.cc:824] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:40.260104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6456
I20260925 13:12:40.260239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:40.260429 392 external_mini_cluster.cc:1428] Setting key 2cea568967a38d38d53fab97bc28d85b
I20260925 13:12:40.260489 6524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: No bootstrap required, opened a new log
I20260925 13:12:40.262615 6524 raft_consensus.cc:348] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260925 13:12:40.262787 6524 raft_consensus.cc:374] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:40.262830 6524 raft_consensus.cc:729] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b382c1aff28e4d7c875ca75c180ace1e, State: Initialized, Role: FOLLOWER
I20260925 13:12:40.262962 6524 consensus_queue.cc:260] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260925 13:12:40.263033 6524 raft_consensus.cc:388] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:40.263062 6524 raft_consensus.cc:482] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:40.263105 6524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:40.263734 6524 raft_consensus.cc:504] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260925 13:12:40.263833 6524 leader_election.cc:302] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b382c1aff28e4d7c875ca75c180ace1e; no voters:
I20260925 13:12:40.263954 6524 leader_election.cc:288] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:40.263991 6529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:40.264127 6529 raft_consensus.cc:686] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 LEADER]: Becoming Leader. State: Replica: b382c1aff28e4d7c875ca75c180ace1e, State: Running, Role: LEADER
I20260925 13:12:40.264196 6524 sys_catalog.cc:564] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:40.264267 6529 consensus_queue.cc:237] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260925 13:12:40.264647 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b382c1aff28e4d7c875ca75c180ace1e. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } }
I20260925 13:12:40.264654 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } }
I20260925 13:12:40.264797 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:40.264804 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:40.265025 6538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:40.265501 6538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:40.267252 6538 catalog_manager.cc:1347] Generated new cluster ID: 7fed07700c7b40d0beffc8ed3689405b
I20260925 13:12:40.267307 6538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:40.277408 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.260870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.0.98.62:42953)
0925 13:12:40.261044 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.261052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.261096 (+ 44us) server_negotiation.cc:407] Connection header received
0925 13:12:40.261852 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.261861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.261919 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.262053 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.262618 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.264209 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.265379 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.266055 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.266165 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.269037 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.269041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.269041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.269052 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.275838 (+ 6786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.276252 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.276255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.276256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.276363 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.276571 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.276571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.276572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.276832 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.276962 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.277090 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.277207 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260925 13:12:40.279760 6538 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:40.280222 6538 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:40.287983 6538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Generated new TSK 0
I20260925 13:12:40.288144 6538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 13:12:40.316504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:40.416618 6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:40.416801 6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:40.416823 6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:40.418045 6550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:40.418092 6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:40.418118 6550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:40.419623 6550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:40.419853 6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:40.420063 6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:40.424755 6558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.424903 6550 server_base.cc:1034] running on GCE node
W20260925 13:12:40.424777 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.424782 6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.425238 6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:40.425421 6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:40.426554 6550 hybrid_clock.cc:648] HybridClock initialized: now 1790341960426534 us; error 34 us; skew 500 ppm
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:40.428857 6550 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:12:40.429207 6550 webserver.cc:460] Webserver started at http://127.0.98.1:45115/ using document root <none> and password file <none>
I20260925 13:12:40.429368 6550 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:40.429404 6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:40.429484 6550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:40.430446 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "3cffbcd807594979b348c51240d2c963"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "cd20a58cd53c15a18b5d429912df8025"
server_key_iv: "39a9623c5ddf7c9b422a44857dbea36e"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.430728 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3cffbcd807594979b348c51240d2c963"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "cd20a58cd53c15a18b5d429912df8025"
server_key_iv: "39a9623c5ddf7c9b422a44857dbea36e"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.431922 6550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:12:40.432552 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.432746 6550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:40.432827 6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "3cffbcd807594979b348c51240d2c963"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "cd20a58cd53c15a18b5d429912df8025"
server_key_iv: "39a9623c5ddf7c9b422a44857dbea36e"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.432876 6550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:40.442989 6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:40.444626 6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:40.444725 6550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:40.444876 6550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:40.445096 6550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:40.445125 6550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.445145 6550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:40.445158 6550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.450682 6550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45195
I20260925 13:12:40.450757 6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45195 every 8 connection(s)
I20260925 13:12:40.451042 6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:40.451336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6550
I20260925 13:12:40.451443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:40.451690 392 external_mini_cluster.cc:1428] Setting key e70a8fa6ff163f8ba17768b338f5aa0f
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:12:40.460062 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.451778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59161 (local address 127.0.98.62:42953)
0925 13:12:40.451825 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.451827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.452069 (+ 242us) server_negotiation.cc:407] Connection header received
0925 13:12:40.452585 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.452587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.452599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.452673 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.453525 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.454096 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.455098 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.455309 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.455390 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.457377 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.457378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.457378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.457386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.458730 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.459125 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.459126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.459126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.459185 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.459438 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.459439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.459440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.459545 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.459623 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.459890 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.459973 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:40.460230 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.451748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:59161)
0925 13:12:40.451921 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.451966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.452492 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.452736 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.452739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.453254 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.453428 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.453433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.454187 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.454188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.454990 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.454991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.455401 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.455402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.455520 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.456003 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.456010 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.457277 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.458837 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.458843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.458845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.459049 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.459271 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.459272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.459272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.459368 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.459614 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.459620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.459766 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.459871 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.460015 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260925 13:12:40.460702 6679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260925 13:12:40.460772 6679 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:40.460933 6679 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 13:12:40.461589 6488 ts_manager.cc:194] Registered new tserver with Master: 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195)
I20260925 13:12:40.462453 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 13:12:40.476157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:40.556133 6684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:40.556352 6684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:40.556375 6684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:40.557472 6684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:40.557507 6684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:40.557533 6684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:40.558947 6684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:40.559122 6684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:40.559284 6684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:40.563979 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.563993 6684 server_base.cc:1034] running on GCE node
W20260925 13:12:40.563920 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.563933 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.564376 6684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:40.564610 6684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:40.565778 6684 hybrid_clock.cc:648] HybridClock initialized: now 1790341960565781 us; error 39 us; skew 500 ppm
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:40.568120 6684 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:12:40.568567 6684 webserver.cc:460] Webserver started at http://127.0.98.2:33957/ using document root <none> and password file <none>
I20260925 13:12:40.568781 6684 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:40.568845 6684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:40.569002 6684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:40.570014 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "741312b211e6423597d6cb2a36643288"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "6a967d4da074c7defe13445aea55f412"
server_key_iv: "f31a563470c9f7232902ffdc4eaa6ebe"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.570356 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "741312b211e6423597d6cb2a36643288"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "6a967d4da074c7defe13445aea55f412"
server_key_iv: "f31a563470c9f7232902ffdc4eaa6ebe"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.571621 6684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:40.572295 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.572458 6684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:40.572551 6684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "741312b211e6423597d6cb2a36643288"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "6a967d4da074c7defe13445aea55f412"
server_key_iv: "f31a563470c9f7232902ffdc4eaa6ebe"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.572628 6684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:40.597064 6684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:40.599542 6684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:40.599711 6684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:40.599956 6684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:40.600273 6684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:40.600329 6684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.600368 6684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:40.600396 6684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.606204 6684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35187
I20260925 13:12:40.606253 6812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35187 every 8 connection(s)
I20260925 13:12:40.606562 6684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:12:40.610888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6684
I20260925 13:12:40.611004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:40.611227 392 external_mini_cluster.cc:1428] Setting key 40bc57678a5eedf4d4396e70c07fde38
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:40.615950 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.607259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44563 (local address 127.0.98.62:42953)
0925 13:12:40.607319 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.607321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.607489 (+ 168us) server_negotiation.cc:407] Connection header received
0925 13:12:40.607978 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.607979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.607988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.608054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.608958 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.609440 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.610479 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.610658 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.610706 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.613105 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.613107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.613107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.613115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.614556 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.615000 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.615001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.615002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.615056 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.615310 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.615311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.615311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.615424 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.615496 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.615762 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.615839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:12:40.616154 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.607222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:44563)
0925 13:12:40.607362 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.607401 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.607900 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.608115 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.608118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.608669 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.608874 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.608878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.609547 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.609548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.610402 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.610404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.610751 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.610753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.610913 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.611638 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.611653 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.613008 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.614673 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.614679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.614681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.614923 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.615128 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.615129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.615130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.615243 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.615488 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.615493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.615631 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.615740 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.615917 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260925 13:12:40.616722 6813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260925 13:12:40.616842 6813 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:40.617012 6813 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260925 13:12:40.617405 6488 ts_manager.cc:194] Registered new tserver with Master: 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187)
I20260925 13:12:40.617936 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44563
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 13:12:40.635833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:40.715978 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:40.716172 6818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:40.716193 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:40.717350 6818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:40.717393 6818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:40.717419 6818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:40.718951 6818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:40.719190 6818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:40.719412 6818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:40.724395 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.724431 6826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.724438 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.724500 6818 server_base.cc:1034] running on GCE node
I20260925 13:12:40.724825 6818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:40.725137 6818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:40.726320 6818 hybrid_clock.cc:648] HybridClock initialized: now 1790341960726269 us; error 58 us; skew 500 ppm
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:40.728924 6818 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:12:40.729409 6818 webserver.cc:460] Webserver started at http://127.0.98.3:43413/ using document root <none> and password file <none>
I20260925 13:12:40.729786 6818 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:40.729874 6818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:40.730070 6818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:40.731294 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "039f42e030729cad4a401bb8890c6197"
server_key_iv: "b6e8e2dbbb04193a49f5a0015d8825cc"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.731729 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "039f42e030729cad4a401bb8890c6197"
server_key_iv: "b6e8e2dbbb04193a49f5a0015d8825cc"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.733166 6818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:40.734090 6833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.734253 6818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.734364 6818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "039f42e030729cad4a401bb8890c6197"
server_key_iv: "b6e8e2dbbb04193a49f5a0015d8825cc"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.734472 6818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:40.744995 6818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:40.746984 6818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:40.747154 6818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:40.747380 6818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:40.747690 6818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:40.747748 6818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.747789 6818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:40.747816 6818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.753152 6818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33745
I20260925 13:12:40.753232 6946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33745 every 8 connection(s)
I20260925 13:12:40.753544 6818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:40.760260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6818
I20260925 13:12:40.760360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:40.760519 392 external_mini_cluster.cc:1428] Setting key 29b568ca1a58b687606a3192a3264bbd
I20260925 13:12:40.762976 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.754303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34555 (local address 127.0.98.62:42953)
0925 13:12:40.754346 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.754348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.754522 (+ 174us) server_negotiation.cc:407] Connection header received
0925 13:12:40.755013 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.755014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.755023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.755082 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.755956 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.756433 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.757403 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.757586 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.757652 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.759821 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.759822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.759823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.759831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.761805 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.762151 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.762152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.762152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.762229 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.762420 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.762421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.762421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.762540 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.762606 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.762848 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.762913 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:40.763056 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.754252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:34555)
0925 13:12:40.754410 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.754450 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.754931 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.755145 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.755148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.755697 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.755885 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.755889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.756509 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.756509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.757303 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.757305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.757705 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.757706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.757856 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.758388 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.758397 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.759712 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.761873 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.761878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.761879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.762084 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.762286 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.762286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.762286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.762367 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.762595 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.762598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.762703 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.762794 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.762907 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260925 13:12:40.763556 6947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260925 13:12:40.763687 6947 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:40.763845 6947 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260925 13:12:40.764183 6488 ts_manager.cc:194] Registered new tserver with Master: 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745)
I20260925 13:12:40.764590 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34555
I20260925 13:12:40.772418 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 13:12:40.796830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42371
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42953,127.0.98.61:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:40.881011 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:40.881258 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:40.881295 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:40.882403 6952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:40.882478 6952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:40.882503 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:40.882541 6952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:40.882575 6952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:40.883991 6952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42953,127.0.98.61:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
W20260925 13:12:40.884140 6952 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42953,127.0.98.61:42371'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 13:12:40.884298 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:40.884531 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:40.889130 6958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.889204 6960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:40.889182 6957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:40.889365 6952 server_base.cc:1034] running on GCE node
I20260925 13:12:40.889600 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:40.889938 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:40.891112 6952 hybrid_clock.cc:648] HybridClock initialized: now 1790341960891089 us; error 51 us; skew 500 ppm
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:40.893589 6952 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 13:12:40.894084 6952 webserver.cc:460] Webserver started at http://127.0.98.61:37035/ using document root <none> and password file <none>
I20260925 13:12:40.894320 6952 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:40.894384 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:40.894515 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:40.895578 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/instance:
uuid: "379d214d810d4d97a82ae6a0df341ac9"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "39f9b8cc6a239d4c69a66fe93007d589"
server_key_iv: "8ed927f85346c6297564e19da0765b25"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.895942 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal/instance:
uuid: "379d214d810d4d97a82ae6a0df341ac9"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "39f9b8cc6a239d4c69a66fe93007d589"
server_key_iv: "8ed927f85346c6297564e19da0765b25"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.897249 6952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:40.898063 6967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.898233 6952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:40.898331 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
uuid: "379d214d810d4d97a82ae6a0df341ac9"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "39f9b8cc6a239d4c69a66fe93007d589"
server_key_iv: "8ed927f85346c6297564e19da0765b25"
server_key_version: "encryptionkey@0"
I20260925 13:12:40.898413 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:40.933466 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:40.935673 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:40.935870 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:40.940202 6952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42371
I20260925 13:12:40.940249 7019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42371 every 8 connection(s)
I20260925 13:12:40.940616 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:12:40.941545 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:40.941841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 6952
I20260925 13:12:40.943948 7020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:12:40.952729 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.944631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.0.98.62:42953)
0925 13:12:40.944714 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.944716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.944854 (+ 138us) server_negotiation.cc:407] Connection header received
0925 13:12:40.944912 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.944913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.944921 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.944963 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.945486 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.946176 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.947177 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.947449 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.947499 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.949587 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.949588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.949588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.949596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.951255 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.951665 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.951666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.951667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.951729 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.951953 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.951954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.951954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.952165 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.952231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.952549 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.952633 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:12:40.952930 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.944589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:45682)
0925 13:12:40.944747 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.944786 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.944856 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.944990 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.944992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.945312 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.945412 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.945416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.946259 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.946259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.947094 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.947096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.947523 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.947524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.947601 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.948060 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.948069 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.949493 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.951351 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.951356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.951357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.951591 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.951798 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.951798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.951798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.951888 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.952230 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.952235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.952388 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.952508 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.952675 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
I20260925 13:12:40.953537 7020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:12:40.953985 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.942463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.98.61:42371)
0925 13:12:40.942699 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.942709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.942762 (+ 53us) server_negotiation.cc:407] Connection header received
0925 13:12:40.943238 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.943246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.943312 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.943469 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.944153 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.945417 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.946530 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.947064 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.947182 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.950390 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.950395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.950396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.950401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.952137 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.952524 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.952526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.952527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.952584 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.953152 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.953153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.953154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.953354 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.953493 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.953766 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.953860 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:12:40.959343 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.953818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54066)
0925 13:12:40.953868 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.953878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.953915 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.954206 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.954207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.954348 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.954387 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.954388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.955079 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.955079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.955552 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.955553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.955926 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.955927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.955986 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.956318 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.956322 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.957365 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.958632 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.958632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.958633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.958784 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.958934 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.958934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.958934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.958973 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.959133 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.959133 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.959162 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.959207 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.959272 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 13:12:40.959394 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.953855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.0.98.61:42371)
0925 13:12:40.954102 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.954104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.954113 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:40.954135 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.954136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.954147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.954204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.954460 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.954976 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.955642 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.955846 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.955926 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.957435 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.957436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.957436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.957441 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.958567 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.958846 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.958847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.958847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.958890 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.959005 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.959006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.959006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.959085 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.959138 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.959220 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.959313 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
I20260925 13:12:40.960565 7020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Bootstrap starting.
I20260925 13:12:40.961429 7020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:40.961700 7020 log.cc:824] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:40.962371 7020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: No bootstrap required, opened a new log
I20260925 13:12:40.963977 7020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } }
I20260925 13:12:40.964219 7020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:40.964267 7020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379d214d810d4d97a82ae6a0df341ac9, State: Initialized, Role: FOLLOWER
I20260925 13:12:40.964394 7020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } }
I20260925 13:12:40.964640 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } } }
I20260925 13:12:40.964704 7020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:40.964726 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:12:40.966624 7038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:12:40.966699 7038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:40.968251 6952 master_runner.cc:428] Detected that this master 379d214d810d4d97a82ae6a0df341ac9 is joining an existing cluster
I20260925 13:12:40.968289 6952 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:42371
I20260925 13:12:40.976140 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.969288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.98.62:42953)
0925 13:12:40.969330 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.969333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.969464 (+ 131us) server_negotiation.cc:407] Connection header received
0925 13:12:40.969583 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.969584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.969593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.969644 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.970157 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.970725 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.971707 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.972008 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.972052 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.973178 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.973180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.973180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.973187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.974854 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975285 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.975286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.975287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.975341 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975540 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.975541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.975541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.975694 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.975757 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.975946 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.976026 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:40.976238 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.969269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.0.98.61:42371)
0925 13:12:40.969305 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.969307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.969549 (+ 242us) server_negotiation.cc:407] Connection header received
0925 13:12:40.969591 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.969592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.969601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.969655 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.970340 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.971039 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.972004 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.972299 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.972377 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.973270 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.973271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.973271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.973277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.975181 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975526 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.975527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.975527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.975572 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975748 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.975749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.975749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.975862 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.975920 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.976107 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.976176 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:12:40.976301 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.969227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:45686)
0925 13:12:40.969387 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.969396 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.969493 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.969737 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.969738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.969959 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.970060 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.970061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.970796 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.970797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.971619 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.971621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.972122 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.972123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.972162 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.972654 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.972660 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.973075 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.974953 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.974954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.974954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.975200 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.975414 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.975415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.975476 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.975783 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.975784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.975823 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.975894 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.975991 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260925 13:12:40.976501 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.969342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54078)
0925 13:12:40.969475 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.969484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.969537 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.970006 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.970006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.970200 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.970247 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.970248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.971139 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.971140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.971927 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.971929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.972377 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.972378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.972433 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.972825 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.972829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.973195 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.975263 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.975264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.975452 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.975638 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.975638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.975639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.975691 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.975941 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.975942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.976010 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.976308 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.976398 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
Sep 25 13:12:40 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:12:40.986380 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.978931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54082)
0925 13:12:40.979071 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:40.979113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:40.979189 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:40.979338 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:40.979342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:40.979601 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:40.979648 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.979650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.980526 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.980527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.981090 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:40.981091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.981534 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.981535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:40.981587 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.982082 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:40.982086 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:40.983604 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:40.985312 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.985318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.985320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.985548 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.985778 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:40.985779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:40.985779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:40.985839 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:40.986092 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:40.986097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:40.986141 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:12:40.986212 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:12:40.986307 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 13:12:40.986413 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:40.978982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.0.98.61:42371)
0925 13:12:40.979016 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:40.979017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:40.979186 (+ 169us) server_negotiation.cc:407] Connection header received
0925 13:12:40.979263 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:40.979264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:40.979274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:40.979326 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:40.979724 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.980448 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.981167 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:40.981459 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:40.981526 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:40.983686 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:40.983688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:40.983688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:40.983694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:40.985229 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.985625 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.985626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.985626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.985708 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:40.985903 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:40.985904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:40.985905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:40.986027 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:40.986102 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:40.986253 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 13:12:40.986322 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:12:40.986982 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:42371: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 42371
}
attrs {
promote: true
}
}
dest_uuid: "b382c1aff28e4d7c875ca75c180ace1e"
cas_config_opid_index: -1
I20260925 13:12:40.987324 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } }
I20260925 13:12:40.987840 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } }
I20260925 13:12:40.988157 6994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:40.988868 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:42371
I20260925 13:12:40.989063 6994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Refusing update from remote peer b382c1aff28e4d7c875ca75c180ace1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 13:12:40.989074 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } }
I20260925 13:12:40.989147 6952 master.cc:537] Master@127.0.98.61:42371 shutting down...
I20260925 13:12:40.989163 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:40.989559 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:12:40.990829 6475 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 13:12:40.991282 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e -> Peer 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61:42371): Couldn't send request to peer 379d214d810d4d97a82ae6a0df341ac9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 13:12:40.992400 6952 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:40.992501 6952 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:40.992544 6952 tablet_replica.cc:330] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: stopping tablet replica
I20260925 13:12:41.005208 6952 master.cc:552] Master@127.0.98.61:42371 shutdown complete.
I20260925 13:12:41.005265 6952 master_runner.cc:305] Clearing existing system tablet
I20260925 13:12:41.005393 6952 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.005438 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.006286 6952 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.006807 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.006956 6952 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.007018 6952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.007042 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
uuid: "379d214d810d4d97a82ae6a0df341ac9"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "39f9b8cc6a239d4c69a66fe93007d589"
server_key_iv: "8ed927f85346c6297564e19da0765b25"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.007818 6952 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:41.009454 6952 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 13:12:41.009522 6952 log.cc:1196] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 13:12:41.009738 6952 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Deleting consensus metadata
I20260925 13:12:41.009871 6952 master_runner.cc:315] Copying system tablet from 127.0.98.62:42953
I20260925 13:12:41.010478 6952 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42953
I20260925 13:12:41.016525 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.010827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.0.98.62:42953)
0925 13:12:41.010879 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.010882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.010939 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:12:41.011021 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.011021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.011032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.011092 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.011424 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.011930 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.012696 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.012900 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.012985 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.014032 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.014041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.014042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.014050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.015387 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.015755 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.015757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.015757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.015818 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.016012 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.016013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.016014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.016132 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.016208 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.016359 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.016434 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:12:41.016579 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.010758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:45694)
0925 13:12:41.010873 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.010885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.010957 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.011094 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.011095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.011288 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.011340 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.011342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.012015 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.012016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.012608 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.012610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.012985 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.012986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.013055 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.013591 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.013595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.013947 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.015486 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.015487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.015487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.015682 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.015883 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.015884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.015885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.015941 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.016199 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.016200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.016244 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.016333 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.016434 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260925 13:12:41.017242 6508 tablet_copy_service.cc:140] P b382c1aff28e4d7c875ca75c180ace1e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 379d214d810d4d97a82ae6a0df341ac9 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45694)
I20260925 13:12:41.017321 6508 tablet_copy_service.cc:161] P b382c1aff28e4d7c875ca75c180ace1e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 379d214d810d4d97a82ae6a0df341ac9 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45694: session id = 379d214d810d4d97a82ae6a0df341ac9-00000000000000000000000000000000
I20260925 13:12:41.018028 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 13:12:41.018671 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 13:12:41.019909 6952 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: tablet copy: Starting download of 0 data blocks...
I20260925 13:12:41.020061 6952 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: tablet copy: Starting download of 1 WAL segments...
I20260925 13:12:41.020928 6952 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 13:12:41.021827 6508 tablet_copy_service.cc:342] P b382c1aff28e4d7c875ca75c180ace1e: Request end of tablet copy session 379d214d810d4d97a82ae6a0df341ac9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45694
I20260925 13:12:41.021890 6508 tablet_copy_service.cc:433] P b382c1aff28e4d7c875ca75c180ace1e: ending tablet copy session 379d214d810d4d97a82ae6a0df341ac9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 379d214d810d4d97a82ae6a0df341ac9
I20260925 13:12:41.024601 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:41.025939 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.026163 6952 server_base.cc:1034] running on GCE node
W20260925 13:12:41.025915 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.025992 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.026415 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:41.026558 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:41.027683 6952 hybrid_clock.cc:648] HybridClock initialized: now 1790341961027684 us; error 35 us; skew 500 ppm
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:41.028991 6952 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 13:12:41.029264 6952 webserver.cc:460] Webserver started at http://127.0.98.61:40003/ using document root <none> and password file <none>
I20260925 13:12:41.029395 6952 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.029448 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.030149 6952 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.031209 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.031366 6952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:41.031414 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
uuid: "379d214d810d4d97a82ae6a0df341ac9"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "39f9b8cc6a239d4c69a66fe93007d589"
server_key_iv: "8ed927f85346c6297564e19da0765b25"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.031473 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.051724 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:41.053396 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:41.056443 6952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42371
I20260925 13:12:41.056483 7122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42371 every 8 connection(s)
I20260925 13:12:41.056771 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:12:41.057441 7123 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:12:41.057947 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Bootstrap starting.
I20260925 13:12:41.060112 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:12:41.060335 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: Bootstrap complete.
I20260925 13:12:41.060524 7123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } }
I20260925 13:12:41.060616 7123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 379d214d810d4d97a82ae6a0df341ac9, State: Initialized, Role: LEARNER
I20260925 13:12:41.060662 7123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } }
I20260925 13:12:41.060832 7123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:41.060807 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } }
I20260925 13:12:41.060869 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:12:41.062201 7137 catalog_manager.cc:1259] Loaded cluster ID: 7fed07700c7b40d0beffc8ed3689405b
I20260925 13:12:41.062268 7137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: loading cluster ID for follower catalog manager: success
I20260925 13:12:41.063228 7137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: acquiring CA information for follower catalog manager: success
I20260925 13:12:41.063498 7137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:12:41.111898 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.105956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.0.98.61:42371)
0925 13:12:41.106088 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.106108 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.106117 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:41.106142 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.106143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.106154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.106246 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.106788 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.107455 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.108189 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.108397 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.108479 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.109628 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.109629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.109630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.109636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.110766 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.111101 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.111102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.111102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.111159 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.111384 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.111384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.111385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.111497 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.111597 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.111699 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.111789 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 13:12:41.136135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46699
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:41.216240 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:41.216468 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:41.216504 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:41.217737 7141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:41.217793 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:41.217818 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:41.217849 7141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:41.217875 7141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:41.219792 7141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:41.220047 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:41.220259 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:41.224838 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.224838 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.224898 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.225169 7141 server_base.cc:1034] running on GCE node
I20260925 13:12:41.225395 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:41.225744 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:41.226933 7141 hybrid_clock.cc:648] HybridClock initialized: now 1790341961226919 us; error 22 us; skew 500 ppm
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:41.229460 7141 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 13:12:41.229938 7141 webserver.cc:460] Webserver started at http://127.0.98.60:44425/ using document root <none> and password file <none>
I20260925 13:12:41.230155 7141 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.230221 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.230405 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:41.231503 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/instance:
uuid: "b231c0376cb149d39523ac690c896539"
format_stamp: "Formatted at 2026-09-25 13:12:41 on dist-test-slave-2d0s"
server_key: "615d2e7a73dfb31be54d4fa04b44b0ba"
server_key_iv: "4a7de9d5925a5abfc6ff2020b4bfca72"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.231843 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal/instance:
uuid: "b231c0376cb149d39523ac690c896539"
format_stamp: "Formatted at 2026-09-25 13:12:41 on dist-test-slave-2d0s"
server_key: "615d2e7a73dfb31be54d4fa04b44b0ba"
server_key_iv: "4a7de9d5925a5abfc6ff2020b4bfca72"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.233125 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:41.233884 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.234102 7141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:41.234202 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
uuid: "b231c0376cb149d39523ac690c896539"
format_stamp: "Formatted at 2026-09-25 13:12:41 on dist-test-slave-2d0s"
server_key: "615d2e7a73dfb31be54d4fa04b44b0ba"
server_key_iv: "4a7de9d5925a5abfc6ff2020b4bfca72"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.234274 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.241377 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:41.243278 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:41.243431 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:41.248272 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46699
I20260925 13:12:41.248317 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46699 every 8 connection(s)
I20260925 13:12:41.248687 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:12:41.249630 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:41.250730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 7141
I20260925 13:12:41.251587 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:12:41.260995 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.252762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.0.98.62:42953)
0925 13:12:41.252804 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.252806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.252932 (+ 126us) server_negotiation.cc:407] Connection header received
0925 13:12:41.252958 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.252959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.252968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.253014 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.253516 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.254087 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.255009 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.255218 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.255303 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.257905 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.257907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.257908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.257915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.259524 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.259972 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.259973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.259973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.260020 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.260284 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.260284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.260284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.260406 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.260470 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.260817 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.260893 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:41.261207 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.252565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:45702)
0925 13:12:41.252760 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.252804 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.252888 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.253084 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.253086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.253339 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.253421 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.253425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.254199 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.254200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.254916 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.254918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.255312 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.255313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.255383 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.255918 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.255927 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.257819 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.259600 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.259606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.259609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.259900 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.260091 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.260092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.260092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.260216 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.260477 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.260481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.260651 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.260771 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.260932 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
I20260925 13:12:41.261770 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.251346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.98.60:46699)
0925 13:12:41.251519 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.251527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.251554 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:12:41.252104 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.252111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.252159 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.252322 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.253015 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.253965 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.254741 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.255057 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.255164 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.258407 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.258414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.258414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.258421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.260334 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.260681 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.260683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.260683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.260740 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.260948 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.260948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.260949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.261178 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.261336 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.261428 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.261632 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260925 13:12:41.261847 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:12:41.268572 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.262355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54112)
0925 13:12:41.262402 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.262412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.262445 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.262574 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.262575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.262699 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.262728 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.262730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.263337 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.263338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.263861 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.263863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.264240 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.264241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.264302 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.264735 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.264739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.265877 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.267581 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.267581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.267582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.267762 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.267960 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.267961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.267961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.268015 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.268283 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.268284 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.268321 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.268403 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.268489 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:12:41.268600 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.262416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.0.98.61:42371)
0925 13:12:41.262460 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.262462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.262471 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:41.262509 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.262510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.262522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.262572 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.262801 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.263259 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.263932 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.264161 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.264231 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.265960 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.265961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.265961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.265968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.267499 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.267839 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.267840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.267840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.267893 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.268101 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.268102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.268102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.268209 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.268279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.268429 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.268501 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:41.269037 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:12:41.274585 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.269252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60716)
0925 13:12:41.269295 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.269304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.269335 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.269446 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.269446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.269541 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.269583 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.269584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.270149 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.270150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.270580 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.270581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.270870 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.270871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.270925 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.271308 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.271329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.272361 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.273857 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.273857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.273857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.274005 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.274160 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.274161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.274161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.274209 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.274388 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.274388 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.274418 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.274466 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.274532 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:12:41.274712 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.269294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.98.60:46699)
0925 13:12:41.269324 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.269326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.269333 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:41.269387 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.269388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.269394 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.269441 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.269636 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.270090 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.270640 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.270810 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.270854 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.272430 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.272430 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.272431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.272436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.273793 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.274075 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.274076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.274076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.274114 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.274250 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.274250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.274250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.274344 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.274385 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.274478 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.274608 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":15}
I20260925 13:12:41.276000 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Bootstrap starting.
I20260925 13:12:41.276788 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:41.277060 7209 log.cc:824] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:41.277813 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: No bootstrap required, opened a new log
I20260925 13:12:41.279176 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } }
I20260925 13:12:41.279299 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:41.279320 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b231c0376cb149d39523ac690c896539, State: Initialized, Role: FOLLOWER
I20260925 13:12:41.279472 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } }
I20260925 13:12:41.279739 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } } }
I20260925 13:12:41.279798 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:41.279853 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:12:41.281437 7225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:12:41.281510 7225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:41.284092 7141 master_runner.cc:428] Detected that this master b231c0376cb149d39523ac690c896539 is joining an existing cluster
I20260925 13:12:41.284137 7141 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46699
I20260925 13:12:41.291857 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.285310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60724)
0925 13:12:41.285717 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.285726 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.285873 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.286272 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.286272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.286422 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.286468 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.286469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.287246 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.287246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.288013 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.288015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.288446 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.288522 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.288964 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.288968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.289272 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.290849 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.290850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.290851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.291007 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.291214 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.291215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.291215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.291279 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.291527 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.291528 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.291565 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.291628 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.291730 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260925 13:12:41.291857 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.285327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.98.60:46699)
0925 13:12:41.285357 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.285359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.285765 (+ 406us) server_negotiation.cc:407] Connection header received
0925 13:12:41.285933 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.285934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.285941 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.285964 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.286536 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.287175 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288091 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.288352 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288386 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.289379 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.289380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.289381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.289386 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.290754 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.291088 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.291089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.291089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.291138 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.291354 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.291355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.291355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.291463 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.291536 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.291656 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.291777 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":14}
I20260925 13:12:41.310098 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.285089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:45704)
0925 13:12:41.285456 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.285467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.285524 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.285722 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.285724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.285936 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.286016 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.286018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.286938 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.286939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.287879 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.287882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.288420 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.288518 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.306300 (+ 17782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.306307 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.306836 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.308905 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.308906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.308906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.309140 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.309359 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.309360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.309433 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.309732 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.309733 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.309780 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.309873 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.309984 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260925 13:12:41.310087 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.285099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.98.62:42953)
0925 13:12:41.285175 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.285177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.285533 (+ 356us) server_negotiation.cc:407] Connection header received
0925 13:12:41.285605 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.285606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.285618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.285701 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.286075 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.286845 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.287980 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.288288 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288330 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.306968 (+ 18638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.306970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.306971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.306980 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.308800 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309229 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.309231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.309231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.309288 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309488 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.309489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.309490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.309632 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.309740 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.309911 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.309984 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 13:12:41.310168 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.285115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.98.61:42371)
0925 13:12:41.285171 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.285173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.285519 (+ 346us) server_negotiation.cc:407] Connection header received
0925 13:12:41.285609 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.285610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.285619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.285645 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.286238 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.286979 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288008 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.288301 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288340 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.306925 (+ 18585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.306927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.306927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.306936 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.308948 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309323 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.309324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.309325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.309382 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309572 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.309573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.309573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.309746 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.309814 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.309984 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.310066 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:12:41.310477 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.285218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54120)
0925 13:12:41.285456 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.285467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.285541 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.285804 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.285805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.286120 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.286170 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.286172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.287057 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.287058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.287919 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.287920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.288424 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.288424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.288483 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.306359 (+ 17876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.306364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.306804 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.309035 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.309036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.309250 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.309447 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.309448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.309448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.309505 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.309818 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.309819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.309863 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.310275 (+ 412us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.310375 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260925 13:12:41.312960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6550
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:12:41.319492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45195
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42953
--webserver_port=45115
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:12:41.320216 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.312512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.0.98.60:46699)
0925 13:12:41.312675 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.312678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.312685 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:41.312716 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.312717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.312726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.312778 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.313098 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.313856 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.314854 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.315194 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.315268 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.317427 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.317428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.317429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.317435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.319001 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.319409 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.319410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.319411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.319467 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.319710 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.319711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.319711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.319824 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.319877 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.320045 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.320124 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 13:12:41.320220 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.312464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60734)
0925 13:12:41.312510 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.312520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.312585 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.312800 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.312801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.312963 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.313011 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.313012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.313946 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.313947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.314776 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.314778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.315274 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.315275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.315344 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.315756 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.315761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.317343 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.319082 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.319083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.319083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.319302 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.319531 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.319532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.319533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.319641 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.319909 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.319910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.319943 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.320021 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.320124 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:12:41.321065 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46699: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b231c0376cb149d39523ac690c896539"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46699
}
attrs {
promote: true
}
}
dest_uuid: "b382c1aff28e4d7c875ca75c180ace1e"
cas_config_opid_index: 5
I20260925 13:12:41.321331 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } }
I20260925 13:12:41.321722 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b231c0376cb149d39523ac690c896539 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.322199 7183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:41.322703 7047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46699
I20260925 13:12:41.322769 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.322827 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.323118 7183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Refusing update from remote peer b382c1aff28e4d7c875ca75c180ace1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 13:12:41.323318 7141 master.cc:537] Master@127.0.98.60:46699 shutting down...
I20260925 13:12:41.323410 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:12:41.324028 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e -> Peer b231c0376cb149d39523ac690c896539 (127.0.98.60:46699): Couldn't send request to peer b231c0376cb149d39523ac690c896539. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 13:12:41.326637 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.321888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54126)
0925 13:12:41.321930 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.321940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.321979 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.322260 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.322261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.322262 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:41.322302 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.322303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.324721 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.324722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.325598 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.325600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.326406 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.326407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.326463 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.326483 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.326527 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.326580 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:12:41.326637 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.322053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.0.98.61:42371)
0925 13:12:41.322091 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.322093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.322100 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:41.322122 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.322122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.322144 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.322200 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:41.322377 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.324627 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.325730 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.326312 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.326405 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.326512 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.326534 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.326570 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:41.326941 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 LEARNER]: Refusing update from remote peer b382c1aff28e4d7c875ca75c180ace1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 13:12:41.327134 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 13:12:41.327590 7141 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:41.327679 7141 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:41.327740 7141 tablet_replica.cc:330] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: stopping tablet replica
I20260925 13:12:41.327647 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b231c0376cb149d39523ac690c896539 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.328783 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.328891 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:12:41.328783 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b382c1aff28e4d7c875ca75c180ace1e. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: true } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.328963 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:12:41.329615 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: attempting to promote NON_VOTER 379d214d810d4d97a82ae6a0df341ac9 to VOTER
I20260925 13:12:41.329857 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } }
I20260925 13:12:41.330405 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 LEARNER]: Refusing update from remote peer b382c1aff28e4d7c875ca75c180ace1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 13:12:41.330657 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.330557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60746)
0925 13:12:41.330597 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.330630 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:12:41.330969 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.330877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60754)
0925 13:12:41.330925 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.330947 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 13:12:41.330992 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 13:12:41.331070 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e -> Peer b231c0376cb149d39523ac690c896539 (127.0.98.60:46699): Couldn't send request to peer b231c0376cb149d39523ac690c896539. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46699: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 13:12:41.331595 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.331740 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.332464 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.332528 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.332612 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.332540 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.332870 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.332963 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:12:41.341351 7141 master.cc:552] Master@127.0.98.60:46699 shutdown complete.
I20260925 13:12:41.341425 7141 master_runner.cc:305] Clearing existing system tablet
I20260925 13:12:41.341589 7141 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.341658 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.342259 7141 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.342725 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.342864 7141 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.342918 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.342947 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
uuid: "b231c0376cb149d39523ac690c896539"
format_stamp: "Formatted at 2026-09-25 13:12:41 on dist-test-slave-2d0s"
server_key: "615d2e7a73dfb31be54d4fa04b44b0ba"
server_key_iv: "4a7de9d5925a5abfc6ff2020b4bfca72"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.343513 7141 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:41.344652 7141 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 13:12:41.344728 7141 log.cc:1196] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 13:12:41.344926 7141 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Deleting consensus metadata
I20260925 13:12:41.345057 7141 master_runner.cc:315] Copying system tablet from 127.0.98.62:42953
I20260925 13:12:41.345667 7141 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42953
I20260925 13:12:41.351979 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.346024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.0.98.62:42953)
0925 13:12:41.346067 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.346069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.346184 (+ 115us) server_negotiation.cc:407] Connection header received
0925 13:12:41.346259 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.346260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.346270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.346324 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.346649 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.347152 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.347893 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.348107 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.348179 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.349174 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.349176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.349176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.349183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.350834 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.351185 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.351186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.351186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.351239 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.351476 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.351477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.351477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.351591 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.351664 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.351814 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.351888 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:41.351984 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.345979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:45720)
0925 13:12:41.346108 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.346118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.346197 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.346328 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.346329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.346519 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.346575 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.346577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.347232 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.347234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.347811 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.347813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.348189 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.348190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.348244 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.348732 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.348737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.349072 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.350932 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.350933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.350933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.351103 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.351333 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.351334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.351334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.351391 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.351656 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.351656 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.351701 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.351789 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.351889 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260925 13:12:41.352485 6508 tablet_copy_service.cc:140] P b382c1aff28e4d7c875ca75c180ace1e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b231c0376cb149d39523ac690c896539 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45720)
I20260925 13:12:41.352573 6508 tablet_copy_service.cc:161] P b382c1aff28e4d7c875ca75c180ace1e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b231c0376cb149d39523ac690c896539 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45720: session id = b231c0376cb149d39523ac690c896539-00000000000000000000000000000000
I20260925 13:12:41.352983 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 13:12:41.353418 7141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 13:12:41.354383 7141 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: tablet copy: Starting download of 0 data blocks...
I20260925 13:12:41.354563 7141 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: tablet copy: Starting download of 1 WAL segments...
I20260925 13:12:41.355226 7141 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 13:12:41.355903 6508 tablet_copy_service.cc:342] P b382c1aff28e4d7c875ca75c180ace1e: Request end of tablet copy session b231c0376cb149d39523ac690c896539-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45720
I20260925 13:12:41.355953 6508 tablet_copy_service.cc:433] P b382c1aff28e4d7c875ca75c180ace1e: ending tablet copy session b231c0376cb149d39523ac690c896539-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b231c0376cb149d39523ac690c896539
I20260925 13:12:41.358136 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:41.359248 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.359287 7246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.359434 7141 server_base.cc:1034] running on GCE node
W20260925 13:12:41.360989 7248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.361183 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:41.361336 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:41.362478 7141 hybrid_clock.cc:648] HybridClock initialized: now 1790341961362468 us; error 23 us; skew 500 ppm
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:41.363786 7141 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 13:12:41.364010 7141 webserver.cc:460] Webserver started at http://127.0.98.60:44949/ using document root <none> and password file <none>
I20260925 13:12:41.364136 7141 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.364194 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.364817 7141 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.365329 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.365449 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.365492 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
uuid: "b231c0376cb149d39523ac690c896539"
format_stamp: "Formatted at 2026-09-25 13:12:41 on dist-test-slave-2d0s"
server_key: "615d2e7a73dfb31be54d4fa04b44b0ba"
server_key_iv: "4a7de9d5925a5abfc6ff2020b4bfca72"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.365556 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.382215 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:41.384068 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:41.387029 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46699
I20260925 13:12:41.387365 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:12:41.388078 7308 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:12:41.388473 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Bootstrap starting.
I20260925 13:12:41.389734 7307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46699 every 8 connection(s)
I20260925 13:12:41.390699 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:12:41.390934 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Bootstrap complete.
I20260925 13:12:41.391127 7308 raft_consensus.cc:348] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } }
I20260925 13:12:41.391220 7308 raft_consensus.cc:729] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b231c0376cb149d39523ac690c896539, State: Initialized, Role: LEARNER
I20260925 13:12:41.391294 7308 consensus_queue.cc:260] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } }
I20260925 13:12:41.391490 7308 sys_catalog.cc:564] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:41.391484 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.391700 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:12:41.392956 7322 catalog_manager.cc:1259] Loaded cluster ID: 7fed07700c7b40d0beffc8ed3689405b
I20260925 13:12:41.393038 7322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: loading cluster ID for follower catalog manager: success
I20260925 13:12:41.394178 7322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: acquiring CA information for follower catalog manager: success
I20260925 13:12:41.394443 7322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 13:12:41.437934 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:41.438215 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:41.438252 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:41.439865 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:41.439924 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:41.439960 7234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:41.442003 7234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45115
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:41.442236 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:41.442474 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:41.447700 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.447682 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.447682 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.448103 7234 server_base.cc:1034] running on GCE node
I20260925 13:12:41.448319 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:41.448559 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:41.449827 7234 hybrid_clock.cc:648] HybridClock initialized: now 1790341961449807 us; error 31 us; skew 500 ppm
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:41.452777 7234 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:12:41.453193 7234 webserver.cc:460] Webserver started at http://127.0.98.1:45115/ using document root <none> and password file <none>
I20260925 13:12:41.453438 7234 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.453523 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.454944 7234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:41.455577 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.455775 7234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:41.455878 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "3cffbcd807594979b348c51240d2c963"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "cd20a58cd53c15a18b5d429912df8025"
server_key_iv: "39a9623c5ddf7c9b422a44857dbea36e"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.456228 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.466717 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:41.468513 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:41.468672 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:41.468907 7234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:41.469209 7234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:41.469242 7234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.469286 7234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:41.469312 7234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.475268 7234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45195
I20260925 13:12:41.475337 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45195 every 8 connection(s)
I20260925 13:12:41.475662 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:41.476217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 7234
I20260925 13:12:41.476382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6684
I20260925 13:12:41.483573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35187
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42953
--webserver_port=33957
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:12:41.489280 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.476825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58573 (local address 127.0.98.61:42371)
0925 13:12:41.476867 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.476869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.477823 (+ 954us) server_negotiation.cc:407] Connection header received
0925 13:12:41.477914 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.477916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.477927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.477959 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.478861 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.479674 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.481585 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.481926 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.481971 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.485737 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.485738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.485739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.485748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.487652 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488150 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.488151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.488151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.488205 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488469 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.488470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.488470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.488628 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.488701 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.489110 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.489190 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:41.489475 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.476695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.98.1:58573)
0925 13:12:41.477766 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.477776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.477847 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.478022 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.478023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.478464 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.478742 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.478748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.479793 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.479794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.481480 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.481483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.482050 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.482051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.482243 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.483102 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.483107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.485593 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.487764 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.487771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.487774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.488076 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.488272 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.488273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.488400 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.488725 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.488731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.488899 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.489023 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.489219 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":27,"thread_start_us":46,"threads_started":1}
I20260925 13:12:41.489886 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.478752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32803 (local address 127.0.98.62:42953)
0925 13:12:41.478798 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.478799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.478883 (+ 84us) server_negotiation.cc:407] Connection header received
0925 13:12:41.478990 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.478991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.479002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.479061 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.479510 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.480329 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.481579 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.481969 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.482012 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.485791 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.485793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.485793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.485802 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.488495 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488921 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.488923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.488923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.488983 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.489216 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.489217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.489217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.489362 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.489409 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.489642 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.489752 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:12:41.489939 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42371
I20260925 13:12:41.490036 7453 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.490130 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.476393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40257 (local address 127.0.98.60:46699)
0925 13:12:41.476927 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.476928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.477070 (+ 142us) server_negotiation.cc:407] Connection header received
0925 13:12:41.477686 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.477688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.477702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.477761 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.479208 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.480204 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.481605 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.481934 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.481988 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.486322 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.486324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.486324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.486332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.488014 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488385 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.488386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.488386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.488440 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488618 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.488619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.488619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.488747 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.488799 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.489852 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.490046 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":53,"threads_started":1}
I20260925 13:12:41.490250 7453 heartbeater.cc:511] Master 127.0.98.61:42371 requested a full tablet report, sending...
I20260925 13:12:41.490474 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.476404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.98.1:40257)
0925 13:12:41.476938 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.476979 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.477557 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.477890 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.477894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.479049 (+ 1155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.479116 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.479117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.480293 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.480294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.481522 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.481524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.482050 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.482051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.482242 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.482994 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.482999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.486250 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.488098 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.488099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.488315 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.488507 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.488508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.488555 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.488821 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.488822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.488899 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.489023 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.490382 (+ 1359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":37,"thread_start_us":74,"threads_started":1}
I20260925 13:12:41.490504 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.478606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:32803)
0925 13:12:41.478825 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.478835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.478919 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.479108 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.479109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.479336 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.479406 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.479408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.480424 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.480425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.481482 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.481484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.482089 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.482090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.482242 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.482944 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.482955 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.485640 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.488597 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.488598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.488598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.488828 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.489063 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.489064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.489064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.489130 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.489474 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.489475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.489522 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.489982 (+ 460us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.490101 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":57,"threads_started":1}
I20260925 13:12:41.490847 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46699
I20260925 13:12:41.490895 7451 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.490962 7451 heartbeater.cc:511] Master 127.0.98.60:46699 requested a full tablet report, sending...
I20260925 13:12:41.490974 7087 ts_manager.cc:194] Registered new tserver with Master: 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195)
I20260925 13:12:41.491233 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260925 13:12:41.491292 7452 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.491266 7264 ts_manager.cc:194] Registered new tserver with Master: 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195)
I20260925 13:12:41.491379 7452 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260925 13:12:41.491725 6488 ts_manager.cc:194] Re-registered known tserver with Master: 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195)
I20260925 13:12:41.492093 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32803
W20260925 13:12:41.570717 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:41.570917 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:41.570942 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:41.572093 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:41.572132 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:41.572158 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:41.573611 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33957
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:41.573892 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:41.574120 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:41.579172 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.579312 7459 server_base.cc:1034] running on GCE node
W20260925 13:12:41.579174 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.579188 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.579607 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:41.579800 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:41.580938 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790341961580919 us; error 34 us; skew 500 ppm
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:41.583297 7459 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:12:41.583694 7459 webserver.cc:460] Webserver started at http://127.0.98.2:33957/ using document root <none> and password file <none>
I20260925 13:12:41.583909 7459 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.583992 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.585263 7459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:41.585879 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.586048 7459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.586146 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "741312b211e6423597d6cb2a36643288"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "6a967d4da074c7defe13445aea55f412"
server_key_iv: "f31a563470c9f7232902ffdc4eaa6ebe"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.586489 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.613265 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:41.615228 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:41.615399 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:41.615655 7459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:41.615962 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:41.616014 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.616055 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:41.616096 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.622062 7459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35187
I20260925 13:12:41.622144 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35187 every 8 connection(s)
I20260925 13:12:41.622467 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:12:41.629601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 7459
I20260925 13:12:41.629796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6818
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:12:41.634406 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.623583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51515 (local address 127.0.98.62:42953)
0925 13:12:41.623638 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.623640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.624273 (+ 633us) server_negotiation.cc:407] Connection header received
0925 13:12:41.624380 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.624382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.624396 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.624455 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.625175 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.625947 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.626834 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.627117 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627179 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.631046 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.631047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.631048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.631073 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.632632 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.633049 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.633050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.633051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.633124 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.633410 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.633412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.633412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.633562 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.633630 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.633972 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.634325 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:12:41.634641 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.623440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:51515)
0925 13:12:41.624218 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.624228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.624297 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.624480 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.624481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.624855 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.625064 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.625070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.626041 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.626042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.626744 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.626745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.627191 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.627295 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.627964 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.627975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.630952 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.632730 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.632736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.632738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.632954 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.633193 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.633194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.633194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.633350 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.633633 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.633638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.633838 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.633949 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.634127 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":9,"thread_start_us":57,"threads_started":1}
I20260925 13:12:41.635257 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260925 13:12:41.635371 7589 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.635568 7589 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260925 13:12:41.635986 6488 ts_manager.cc:194] Re-registered known tserver with Master: 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187)
I20260925 13:12:41.636138 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.623231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52313 (local address 127.0.98.61:42371)
0925 13:12:41.623269 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.623270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.623594 (+ 324us) server_negotiation.cc:407] Connection header received
0925 13:12:41.624160 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.624162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.624175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.624243 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.625161 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.625651 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627013 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.627295 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627343 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.633085 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.633087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.633087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.633095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.634871 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.635253 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.635254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.635255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.635309 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.635523 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.635524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.635525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.635662 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.635728 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.635900 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.636002 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:12:41.636328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33745
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42953
--webserver_port=43413
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:12:41.636490 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.623172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.98.2:52313)
0925 13:12:41.623534 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.623543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.624047 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.624330 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.624349 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.624856 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.625064 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.625068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.625835 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.625836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.626918 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.626919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.627363 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.627423 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.627979 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.627983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.632170 (+ 4187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.634955 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.634956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.634956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.635173 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.635373 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.635374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.635374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.635457 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.635738 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.635738 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.635783 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.636254 (+ 471us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.636362 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260925 13:12:41.636538 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51515
I20260925 13:12:41.636996 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42371
I20260925 13:12:41.637048 7590 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.637254 7590 heartbeater.cc:511] Master 127.0.98.61:42371 requested a full tablet report, sending...
I20260925 13:12:41.637578 7087 ts_manager.cc:194] Registered new tserver with Master: 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187)
I20260925 13:12:41.639134 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.623172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.98.2:55455)
0925 13:12:41.623328 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.623372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.624041 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.624329 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.624334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.624855 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.625064 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.625070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.626025 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.626026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.626916 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.626918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.627718 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.627794 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.628169 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.628173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.629762 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.637660 (+ 7898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.637661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.637662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.637898 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.638126 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.638127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.638127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.638183 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.638435 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.638436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.638487 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.638915 (+ 428us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.639025 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":35,"thread_start_us":80,"threads_started":1}
I20260925 13:12:41.639135 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.623166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55455 (local address 127.0.98.60:46699)
0925 13:12:41.623210 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.623212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.623468 (+ 256us) server_negotiation.cc:407] Connection header received
0925 13:12:41.624161 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.624162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.624178 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.624243 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.625173 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.625945 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627385 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.627643 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.627713 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.629860 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.629861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.629862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.629869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.632984 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.637986 (+ 5002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.637987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.637988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.638057 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.638254 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.638255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.638255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.638369 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.638444 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.638952 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.639038 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:12:41.639573 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46699
I20260925 13:12:41.639627 7588 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.639690 7588 heartbeater.cc:511] Master 127.0.98.60:46699 requested a full tablet report, sending...
I20260925 13:12:41.639933 7264 ts_manager.cc:194] Registered new tserver with Master: 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187)
W20260925 13:12:41.717397 7595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:41.717651 7595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:41.717687 7595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:41.718824 7595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:41.718896 7595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:41.718950 7595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:41.720562 7595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43413
--tserver_master_addrs=127.0.98.62:42953,127.0.98.61:42371,127.0.98.60:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:41.720817 7595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:41.721040 7595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:41.725935 7603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.725986 7600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:41.725934 7601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:41.726153 7595 server_base.cc:1034] running on GCE node
I20260925 13:12:41.726392 7595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:41.726619 7595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:41.727795 7595 hybrid_clock.cc:648] HybridClock initialized: now 1790341961727773 us; error 33 us; skew 500 ppm
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:41.730250 7595 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:12:41.730655 7595 webserver.cc:460] Webserver started at http://127.0.98.3:43413/ using document root <none> and password file <none>
I20260925 13:12:41.730901 7595 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:41.730980 7595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:41.732383 7595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:41.732970 7610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.733155 7595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.733268 7595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4"
format_stamp: "Formatted at 2026-09-25 13:12:40 on dist-test-slave-2d0s"
server_key: "039f42e030729cad4a401bb8890c6197"
server_key_iv: "b6e8e2dbbb04193a49f5a0015d8825cc"
server_key_version: "encryptionkey@0"
I20260925 13:12:41.733614 7595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:41.739705 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.736834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.0.98.60:46699)
0925 13:12:41.736898 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.736900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.736911 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:41.736991 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.736993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.737014 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.737112 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:41.737263 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.737813 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.739070 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.739432 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.739488 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.739529 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.739627 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.739657 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:12:41.739725 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.736768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60770)
0925 13:12:41.736839 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.736864 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.736909 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.737125 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.737126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.737129 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:41.737176 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.737178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.737877 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.737878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.738988 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.738990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.739505 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.739505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.739562 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.739566 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.739612 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.739654 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 13:12:41.740015 7275 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 13:12:41.740175 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b382c1aff28e4d7c875ca75c180ace1e. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.740278 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:12:41.746079 7595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:41.748292 7595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:41.748402 7595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:41.748579 7595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:41.748884 7595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:41.748919 7595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.748940 7595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:41.748987 7595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:41.754827 7595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33745
I20260925 13:12:41.754927 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33745 every 8 connection(s)
I20260925 13:12:41.755242 7595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:12:41.762131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 7595
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 13:12:41 dist-test-slave-2d0s krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:12:41.766908 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.755934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33505 (local address 127.0.98.60:46699)
0925 13:12:41.755975 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.755977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.756691 (+ 714us) server_negotiation.cc:407] Connection header received
0925 13:12:41.757288 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.757289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.757306 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.757336 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.758285 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.759071 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760200 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.760417 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760447 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.763355 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.763356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.763356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.763364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.765230 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.765779 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.765781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.765781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.765854 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766152 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.766152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.766153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.766289 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.766342 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.766724 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.766805 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:12:41.767057 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.756250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:47945)
0925 13:12:41.756371 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.756380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.757177 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.757752 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.757754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.757922 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.758169 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.758173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.758868 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.758871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.759746 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.759748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.760133 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.760290 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.760916 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.760924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.762636 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.765731 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.765732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.765732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.765946 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.766191 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.766192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.766260 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.766506 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.766507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.766554 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.766663 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.766840 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":44,"thread_start_us":61,"threads_started":1}
I20260925 13:12:41.767338 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.755941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.98.3:33505)
0925 13:12:41.756116 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.756163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.757162 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.757433 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.757433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.757917 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.758170 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.758176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.759195 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.759195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.760102 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.760104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.760548 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.760609 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.761110 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.761115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.763257 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.765323 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.765329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.765331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.765689 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.765925 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.765926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.765926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.766071 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.766377 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.766383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.766554 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.767092 (+ 538us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.767204 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":33,"thread_start_us":90,"threads_started":1}
I20260925 13:12:41.767566 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.756305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47945 (local address 127.0.98.62:42953)
0925 13:12:41.756351 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.756353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.756751 (+ 398us) server_negotiation.cc:407] Connection header received
0925 13:12:41.757267 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.757268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.757281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.757336 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.758262 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.758760 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.759839 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.760038 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760123 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.762732 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.762734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.762734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.762743 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.765586 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766066 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.766068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.766068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.766125 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766319 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.766320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.766320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.766435 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.766485 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.767378 (+ 893us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.767462 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:41.767782 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46699
I20260925 13:12:41.767879 7724 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.767949 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260925 13:12:41.767993 7725 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.768044 7724 heartbeater.cc:511] Master 127.0.98.60:46699 requested a full tablet report, sending...
I20260925 13:12:41.768052 7725 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260925 13:12:41.768440 6488 ts_manager.cc:194] Re-registered known tserver with Master: 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745)
I20260925 13:12:41.768472 7264 ts_manager.cc:194] Registered new tserver with Master: 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745)
I20260925 13:12:41.768918 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47945
I20260925 13:12:41.769224 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.755991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57563 (local address 127.0.98.61:42371)
0925 13:12:41.756030 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.756032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.756729 (+ 697us) server_negotiation.cc:407] Connection header received
0925 13:12:41.757288 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.757290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.757302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.757337 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.758279 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.759076 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760230 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.760530 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760622 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.764320 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.764322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.764322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.764330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.766293 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766705 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.766706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.766706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.766760 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766945 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.766946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.766947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.767089 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.767129 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.769051 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.769123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:12:41.769769 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.755941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.98.3:57563)
0925 13:12:41.756116 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.756163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.757162 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.757424 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.757429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.757917 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:41.758170 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.758176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.759169 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.759169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.760102 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.760104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.760623 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.760624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.760695 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.761310 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:41.761315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:41.763876 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:41.766391 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.766392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.766625 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.766831 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:41.766832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:41.766832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:41.766882 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:41.768899 (+ 2017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:41.768899 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:41.768933 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.769011 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.769116 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":116,"thread_start_us":99,"threads_started":1}
I20260925 13:12:41.770850 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42371
I20260925 13:12:41.771188 7726 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:41.771291 7726 heartbeater.cc:511] Master 127.0.98.61:42371 requested a full tablet report, sending...
I20260925 13:12:41.771848 7087 ts_manager.cc:194] Registered new tserver with Master: 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745)
I20260925 13:12:41.773926 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.763995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.98.60:46699)
0925 13:12:41.766979 (+ 2984us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.766981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.766987 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:12:41.767003 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.767004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.767013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.767066 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.767772 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.768451 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.769397 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.769680 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.769748 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.771146 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.771148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.771148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.771154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.772664 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.773072 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.773073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.773073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.773133 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.773361 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.773361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.773362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.773478 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.773571 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.773762 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.773837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2969,"thread_start_us":58,"threads_started":1}
I20260925 13:12:41.775012 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.764701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.0.98.61:42371)
0925 13:12:41.764815 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.764817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.764825 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:41.764846 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.764846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.764859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.765781 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.765818 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.766592 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.770284 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.770613 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.770692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.772079 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.772081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.772081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.772088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.773839 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.774164 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.774165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.774165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.774219 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.774453 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.774454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.774454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.774606 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.774704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.774883 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.774950 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260925 13:12:41.777938 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.763480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.98.62:42953)
0925 13:12:41.769231 (+ 5751us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.769233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.769242 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:41.769262 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.769263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.769274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.772099 (+ 2825us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:41.772141 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.772746 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.773777 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.774093 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.774381 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.775579 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:41.775580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:41.775581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:41.775586 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:41.777010 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.777275 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.777276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.777276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.777318 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:41.777454 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:41.777454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:41.777455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:41.777534 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:41.777595 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.777764 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.777824 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":74,"threads_started":1}
I20260925 13:12:41.779124 6488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:41.780633 6488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:41.781337 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: true } } }
I20260925 13:12:41.781430 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.782480 7048 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: attempting to promote NON_VOTER b231c0376cb149d39523ac690c896539 to VOTER
I20260925 13:12:41.782613 7048 consensus_queue.cc:237] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:41.782927 7275 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 LEARNER]: Refusing update from remote peer b382c1aff28e4d7c875ca75c180ace1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 13:12:41.783118 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 13:12:41.783198 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Refusing update from remote peer b382c1aff28e4d7c875ca75c180ace1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 13:12:41.783361 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 13:12:41.783851 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b231c0376cb149d39523ac690c896539 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.783987 7275 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b231c0376cb149d39523ac690c896539 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.784127 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b231c0376cb149d39523ac690c896539 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.784902 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.784924 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.785023 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.785048 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.785148 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:12:41.785125 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.785209 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.784988 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:41.786077 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b382c1aff28e4d7c875ca75c180ace1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:41.786160 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:12:41.790203 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.786251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35187 (local address 127.0.0.1:49796)
0925 13:12:41.786713 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.786724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.786772 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.787007 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.787008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.787010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:41.787061 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.787062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.788183 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.788184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.788963 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.788965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.789967 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.789968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.790045 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.790049 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.790094 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.790134 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":87,"threads_started":1}
I20260925 13:12:41.790534 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.786142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45195 (local address 127.0.0.1:35312)
0925 13:12:41.786198 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.786218 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.786263 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.786608 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.786609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.786611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:41.786652 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.786654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.787922 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.787923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.789022 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.789024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.789948 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.789949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.790008 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.790011 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.790043 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.790461 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260925 13:12:41.790664 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.786377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.0.98.2:35187)
0925 13:12:41.786615 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.786620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.786825 (+ 205us) server_negotiation.cc:407] Connection header received
0925 13:12:41.786865 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.786867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.786881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.786947 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:41.787130 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.788087 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.789045 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.789876 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.789954 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.790496 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.790525 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.790571 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260925 13:12:41.790977 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.786247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.0.98.1:45195)
0925 13:12:41.786452 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.786457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.786467 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:41.786508 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.786510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.786527 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.786600 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:41.786731 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.787834 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.789098 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.789844 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.790701 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.790797 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.790826 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.790901 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260925 13:12:41.790979 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.786592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33745 (local address 127.0.0.1:40950)
0925 13:12:41.787124 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:41.787133 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:41.787182 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:41.787434 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:41.787435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:41.787437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:41.787476 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.787477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.788613 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.788614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.789623 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:41.789625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.790754 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.790755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:41.790816 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.790820 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:41.790874 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:12:41.790915 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":73,"threads_started":1}
I20260925 13:12:41.791056 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:41.786838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.0.98.3:33745)
0925 13:12:41.786992 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:41.786996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:41.787242 (+ 246us) server_negotiation.cc:407] Connection header received
0925 13:12:41.787285 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:41.787287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:41.787302 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:41.787373 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:41.787577 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.788523 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.789759 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:41.790557 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:41.790648 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:41.790723 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:41.790921 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 13:12:41.790959 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260925 13:12:41.791460 7522 tablet_service.cc:1467] Processing CreateTablet for tablet b94aeee7443b42aa80e1ebbcf5a9de93 (DEFAULT_TABLE table=test-table [id=cfb56c6194a24e9ca4d0a0c5355c9742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:41.791838 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94aeee7443b42aa80e1ebbcf5a9de93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:41.791729 7658 tablet_service.cc:1467] Processing CreateTablet for tablet b94aeee7443b42aa80e1ebbcf5a9de93 (DEFAULT_TABLE table=test-table [id=cfb56c6194a24e9ca4d0a0c5355c9742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:41.791934 7385 tablet_service.cc:1467] Processing CreateTablet for tablet b94aeee7443b42aa80e1ebbcf5a9de93 (DEFAULT_TABLE table=test-table [id=cfb56c6194a24e9ca4d0a0c5355c9742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:41.792126 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94aeee7443b42aa80e1ebbcf5a9de93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:41.792240 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94aeee7443b42aa80e1ebbcf5a9de93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:41.794998 7753 tablet_bootstrap.cc:492] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Bootstrap starting.
I20260925 13:12:41.795356 7754 tablet_bootstrap.cc:492] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Bootstrap starting.
I20260925 13:12:41.795806 7753 tablet_bootstrap.cc:654] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:41.795907 7754 tablet_bootstrap.cc:654] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:41.796136 7753 log.cc:824] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:41.796232 7754 log.cc:824] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:41.796517 7755 tablet_bootstrap.cc:492] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Bootstrap starting.
I20260925 13:12:41.796854 7753 tablet_bootstrap.cc:492] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: No bootstrap required, opened a new log
I20260925 13:12:41.796937 7753 ts_tablet_manager.cc:1397] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:41.796929 7754 tablet_bootstrap.cc:492] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: No bootstrap required, opened a new log
I20260925 13:12:41.797400 7754 ts_tablet_manager.cc:1397] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:41.797608 7755 tablet_bootstrap.cc:654] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:41.797922 7755 log.cc:824] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:41.798832 7755 tablet_bootstrap.cc:492] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: No bootstrap required, opened a new log
I20260925 13:12:41.798903 7755 ts_tablet_manager.cc:1397] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:12:41.799175 7753 raft_consensus.cc:348] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:41.799326 7753 raft_consensus.cc:374] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:41.799386 7753 raft_consensus.cc:729] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4033ae0eacbd4a34be3132ac0cf8f7e4, State: Initialized, Role: FOLLOWER
I20260925 13:12:41.799528 7753 consensus_queue.cc:260] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:41.799801 7753 ts_tablet_manager.cc:1428] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:41.799913 7725 heartbeater.cc:503] Master 127.0.98.62:42953 was elected leader, sending a full tablet report...
I20260925 13:12:41.800184 7754 raft_consensus.cc:348] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:41.800338 7754 raft_consensus.cc:374] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:41.800384 7754 raft_consensus.cc:729] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741312b211e6423597d6cb2a36643288, State: Initialized, Role: FOLLOWER
I20260925 13:12:41.800540 7754 consensus_queue.cc:260] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:41.800854 7754 ts_tablet_manager.cc:1428] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:12:41.801273 7589 heartbeater.cc:503] Master 127.0.98.62:42953 was elected leader, sending a full tablet report...
I20260925 13:12:41.802091 7755 raft_consensus.cc:348] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:41.802294 7755 raft_consensus.cc:374] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:41.802353 7755 raft_consensus.cc:729] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffbcd807594979b348c51240d2c963, State: Initialized, Role: FOLLOWER
I20260925 13:12:41.802457 7755 consensus_queue.cc:260] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:41.802696 7755 ts_tablet_manager.cc:1428] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 13:12:41.802747 7452 heartbeater.cc:503] Master 127.0.98.62:42953 was elected leader, sending a full tablet report...
W20260925 13:12:41.873181 7591 tablet.cc:2369] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:12:41.978880 7454 tablet.cc:2369] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:12:42.007061 7727 tablet.cc:2369] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:42.172418 7760 raft_consensus.cc:482] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:42.172547 7760 raft_consensus.cc:504] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:42.172910 7760 leader_election.cc:288] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195), 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745)
I20260925 13:12:42.175896 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.173155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42865 (local address 127.0.98.1:45195)
0925 13:12:42.173202 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:42.173205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:42.173302 (+ 97us) server_negotiation.cc:407] Connection header received
0925 13:12:42.173385 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:42.173385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:42.173398 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:42.173474 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:42.173643 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.174165 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.175163 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.175528 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.175620 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.175631 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:42.175770 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 13:12:42.175806 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:42.175895 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.173145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45003 (local address 127.0.98.3:33745)
0925 13:12:42.173193 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:42.173195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:42.173296 (+ 101us) server_negotiation.cc:407] Connection header received
0925 13:12:42.173399 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:42.173399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:42.173417 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:42.173478 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:42.173651 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.174165 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.175162 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.175536 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.175627 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.175637 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:42.175786 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 13:12:42.175826 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:42.175915 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.173075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33745 (local address 127.0.98.2:45003)
0925 13:12:42.173203 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:42.173213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:42.173312 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:42.173483 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:42.173483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:42.173486 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:42.173571 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:42.173579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.174255 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.174256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:42.175062 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:42.175064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.175628 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.175629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:42.175708 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.175711 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:42.175780 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:12:42.175823 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":56,"thread_start_us":62,"threads_started":1}
I20260925 13:12:42.176038 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.173091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45195 (local address 127.0.98.2:42865)
0925 13:12:42.173223 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:42.173233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:42.173318 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:42.173479 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:42.173480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:42.173482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:42.173559 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:42.173561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.174250 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.174251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:42.175067 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:42.175069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.175621 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.175622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:42.175693 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.175696 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:42.175765 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:12:42.175811 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":18,"thread_start_us":51,"threads_started":1}
I20260925 13:12:42.176215 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94aeee7443b42aa80e1ebbcf5a9de93" candidate_uuid: "741312b211e6423597d6cb2a36643288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" is_pre_election: true
I20260925 13:12:42.176366 7678 raft_consensus.cc:2454] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741312b211e6423597d6cb2a36643288 in term 0.
I20260925 13:12:42.176424 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94aeee7443b42aa80e1ebbcf5a9de93" candidate_uuid: "741312b211e6423597d6cb2a36643288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffbcd807594979b348c51240d2c963" is_pre_election: true
I20260925 13:12:42.176600 7405 raft_consensus.cc:2454] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741312b211e6423597d6cb2a36643288 in term 0.
I20260925 13:12:42.176591 7478 leader_election.cc:302] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4033ae0eacbd4a34be3132ac0cf8f7e4, 741312b211e6423597d6cb2a36643288; no voters:
I20260925 13:12:42.176734 7760 raft_consensus.cc:2781] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:42.176805 7760 raft_consensus.cc:482] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:42.176829 7760 raft_consensus.cc:3037] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:42.177492 7760 raft_consensus.cc:504] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:42.177800 7760 leader_election.cc:288] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [CANDIDATE]: Term 1 election: Requested vote from peers 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195), 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745)
I20260925 13:12:42.177793 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94aeee7443b42aa80e1ebbcf5a9de93" candidate_uuid: "741312b211e6423597d6cb2a36643288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4"
I20260925 13:12:42.177875 7678 raft_consensus.cc:3037] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:42.177897 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94aeee7443b42aa80e1ebbcf5a9de93" candidate_uuid: "741312b211e6423597d6cb2a36643288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffbcd807594979b348c51240d2c963"
I20260925 13:12:42.177973 7405 raft_consensus.cc:3037] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:42.178784 7678 raft_consensus.cc:2454] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741312b211e6423597d6cb2a36643288 in term 1.
I20260925 13:12:42.178843 7405 raft_consensus.cc:2454] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741312b211e6423597d6cb2a36643288 in term 1.
I20260925 13:12:42.178983 7477 leader_election.cc:302] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffbcd807594979b348c51240d2c963, 741312b211e6423597d6cb2a36643288; no voters:
I20260925 13:12:42.179091 7760 raft_consensus.cc:2781] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:42.179229 7760 raft_consensus.cc:686] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 1 LEADER]: Becoming Leader. State: Replica: 741312b211e6423597d6cb2a36643288, State: Running, Role: LEADER
I20260925 13:12:42.179320 7760 consensus_queue.cc:237] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:42.180141 6488 catalog_manager.cc:5495] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 741312b211e6423597d6cb2a36643288 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "741312b211e6423597d6cb2a36643288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } health_report { overall_health: HEALTHY } } }
I20260925 13:12:42.229978 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.227619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.0.98.2:35187)
0925 13:12:42.227706 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:42.227708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:42.227718 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:42.227759 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:42.227760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:42.227780 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:42.227847 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:42.227995 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.228454 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.229182 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.229374 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.229421 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.229553 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:42.229654 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:42.229805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:42.229856 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:12:42.229877 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 13:12:42.231809 7405 raft_consensus.cc:1261] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 1 FOLLOWER]: Refusing update from remote peer 741312b211e6423597d6cb2a36643288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:42.231809 7678 raft_consensus.cc:1261] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Refusing update from remote peer 741312b211e6423597d6cb2a36643288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:42.232180 7760 consensus_queue.cc:1035] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:42.232378 7764 consensus_queue.cc:1035] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:42.235445 7769 mvcc.cc:205] Tried to move back new op lower bound from 7333240677298810880 to 7333240677087125504. Current Snapshot: MvccSnapshot[applied={T|T < 7333240677298810880}]
I20260925 13:12:42.237758 6488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:42.237881 6488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:42.239430 6488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b382c1aff28e4d7c875ca75c180ace1e: Sending DeleteTablet for 3 replicas of tablet b94aeee7443b42aa80e1ebbcf5a9de93
I20260925 13:12:42.239768 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet b94aeee7443b42aa80e1ebbcf5a9de93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35312
I20260925 13:12:42.239775 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet b94aeee7443b42aa80e1ebbcf5a9de93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49796
I20260925 13:12:42.240001 7777 tablet_replica.cc:330] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: stopping tablet replica
I20260925 13:12:42.240080 7776 tablet_replica.cc:330] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: stopping tablet replica
I20260925 13:12:42.240136 7777 raft_consensus.cc:2229] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:42.240249 7776 raft_consensus.cc:2229] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:42.240276 7777 raft_consensus.cc:2258] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:42.240444 7776 raft_consensus.cc:2258] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:42.240739 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet b94aeee7443b42aa80e1ebbcf5a9de93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260925 13:12:42.240865 7777 ts_tablet_manager.cc:1905] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:42.241377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6456
I20260925 13:12:42.243058 7777 ts_tablet_manager.cc:1918] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:42.243137 7777 log.cc:1196] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/b94aeee7443b42aa80e1ebbcf5a9de93
I20260925 13:12:42.243444 7777 ts_tablet_manager.cc:1939] T b94aeee7443b42aa80e1ebbcf5a9de93 P 741312b211e6423597d6cb2a36643288: Deleting consensus metadata
W20260925 13:12:42.247278 7476 connection.cc:535] server connection from 127.0.0.1:49796 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 13:12:42.248530 7776 ts_tablet_manager.cc:1905] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:42.249771 7779 tablet_replica.cc:330] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: stopping tablet replica
I20260925 13:12:42.249899 7779 raft_consensus.cc:2229] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:42.250015 7779 raft_consensus.cc:2258] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:42.250298 7776 ts_tablet_manager.cc:1918] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:42.250373 7776 log.cc:1196] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/b94aeee7443b42aa80e1ebbcf5a9de93
I20260925 13:12:42.250706 7776 ts_tablet_manager.cc:1939] T b94aeee7443b42aa80e1ebbcf5a9de93 P 3cffbcd807594979b348c51240d2c963: Deleting consensus metadata
W20260925 13:12:42.251027 7341 connection.cc:441] server connection from 127.0.0.1:35312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35312 (request call id 1) could send its response
I20260925 13:12:42.252146 7779 ts_tablet_manager.cc:1905] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:42.253882 7779 ts_tablet_manager.cc:1918] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:42.254024 7779 log.cc:1196] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/b94aeee7443b42aa80e1ebbcf5a9de93
I20260925 13:12:42.254410 7779 ts_tablet_manager.cc:1939] T b94aeee7443b42aa80e1ebbcf5a9de93 P 4033ae0eacbd4a34be3132ac0cf8f7e4: Deleting consensus metadata
W20260925 13:12:42.254791 7612 connection.cc:441] server connection from 127.0.0.1:40950 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40950 (request call id 1) could send its response
I20260925 13:12:42.256479 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.240713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.0.98.60:46699)
0925 13:12:42.240751 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:42.240753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:42.240760 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:42.240779 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:42.240780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:42.240790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:42.240839 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:42.249066 (+ 8227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.249602 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.250692 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.250941 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.251030 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.252704 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:42.252706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:42.252706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:42.252714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:42.254550 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:42.254941 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:42.254943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:42.254943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:42.255000 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:42.255180 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:42.255181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:42.255181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:42.255584 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:42.255965 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:42.256263 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 13:12:42.256349 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:42.259435 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.251653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.0.98.60:46699)
0925 13:12:42.251703 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:42.251705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:42.251960 (+ 255us) server_negotiation.cc:407] Connection header received
0925 13:12:42.252059 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:42.252060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:42.252072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:42.252109 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:42.252825 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.253529 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.254659 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.254938 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.254984 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.256383 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:42.256385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:42.256385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:42.256391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:42.258200 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:42.258522 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:42.258523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:42.258524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:42.258580 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:42.258842 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:42.258843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:42.258843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:42.258962 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:42.259088 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:42.259205 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 13:12:42.259311 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:12:42.260319 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:42.251668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.98.61:42371)
0925 13:12:42.251738 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:42.251740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:42.251824 (+ 84us) server_negotiation.cc:407] Connection header received
0925 13:12:42.251928 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:42.251929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:42.251946 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:42.251977 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:42.253039 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.253876 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.254956 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:42.255291 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:42.255335 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:42.256730 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:42.256731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:42.256732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:42.256739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:42.258790 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:42.259411 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:42.259412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:42.259412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:42.259476 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:42.259703 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:42.259704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:42.259704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:42.259841 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:42.259911 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:42.260134 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 13:12:42.260258 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 13:12:43.236302 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.235937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:59979)
0925 13:12:43.236154 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.236248 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
W20260925 13:12:43.236595 7725 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42953 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.237542 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.237314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:34755)
0925 13:12:43.237421 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.237495 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
W20260925 13:12:43.237761 7452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42953 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.237869 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.237651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:35139)
0925 13:12:43.237757 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.237820 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
W20260925 13:12:43.238029 7589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42953 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.739550 7789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b382c1aff28e4d7c875ca75c180ace1e)
I20260925 13:12:43.739696 7789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:43.739782 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b382c1aff28e4d7c875ca75c180ace1e)
I20260925 13:12:43.739928 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:43.740006 7789 leader_election.cc:288] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953), b231c0376cb149d39523ac690c896539 (127.0.98.60:46699)
I20260925 13:12:43.740198 7790 leader_election.cc:288] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953), 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61:42371)
I20260925 13:12:43.740533 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.740156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46218)
0925 13:12:43.740441 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.740490 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":59,"threads_started":1}
I20260925 13:12:43.740531 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.740319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46224)
0925 13:12:43.740441 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.740490 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":49,"threads_started":1}
W20260925 13:12:43.740721 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
W20260925 13:12:43.741024 7073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.741341 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.741234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46234)
0925 13:12:43.741282 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.741305 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:12:43.741508 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.741426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46240)
0925 13:12:43.741471 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.741488 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 13:12:43.741499 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
W20260925 13:12:43.741633 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.743788 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.740458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.0.98.61:42371)
0925 13:12:43.740592 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:43.740594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:43.740604 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:43.740684 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:43.740685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:43.740694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:43.740769 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:43.740932 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.741752 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.743121 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.743462 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.743547 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:43.743559 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:43.743681 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 13:12:43.743727 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":47,"threads_started":1}
I20260925 13:12:43.743804 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.740341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42371 (local address 127.0.0.1:54158)
0925 13:12:43.740546 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.740557 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:43.740614 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:43.740787 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:43.740788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:43.740790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:43.740851 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:43.740852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.741851 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.741852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:43.743027 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:43.743030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.743558 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.743559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:43.743615 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:43.743618 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:43.743667 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:12:43.743736 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":37,"threads_started":1}
I20260925 13:12:43.743894 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.740334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46699 (local address 127.0.0.1:60802)
0925 13:12:43.740711 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.740720 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:43.740767 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:43.740950 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:43.740951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:43.740953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:43.740997 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:43.740999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.741909 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.741909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:43.743027 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:43.743030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.743615 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.743615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:43.743673 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:43.743677 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:43.743730 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 13:12:43.743763 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":51,"threads_started":1}
I20260925 13:12:43.744060 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.740510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.0.98.60:46699)
0925 13:12:43.740638 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:43.740640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:43.740817 (+ 177us) server_negotiation.cc:407] Connection header received
0925 13:12:43.740839 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:43.740840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:43.740849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:43.740913 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:43.741063 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.741812 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.743126 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:43.743465 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:43.743597 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:43.743607 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:43.743743 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 13:12:43.743776 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":37,"threads_started":1}
I20260925 13:12:43.744048 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b231c0376cb149d39523ac690c896539" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "379d214d810d4d97a82ae6a0df341ac9" is_pre_election: true
I20260925 13:12:43.744158 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b231c0376cb149d39523ac690c896539 in term 1.
I20260925 13:12:43.744192 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "379d214d810d4d97a82ae6a0df341ac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b231c0376cb149d39523ac690c896539" is_pre_election: true
I20260925 13:12:43.744274 7275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379d214d810d4d97a82ae6a0df341ac9 in term 1.
I20260925 13:12:43.744295 7259 leader_election.cc:302] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 379d214d810d4d97a82ae6a0df341ac9, b231c0376cb149d39523ac690c896539; no voters: b382c1aff28e4d7c875ca75c180ace1e
I20260925 13:12:43.744469 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 13:12:43.744515 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b382c1aff28e4d7c875ca75c180ace1e)
I20260925 13:12:43.744483 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 379d214d810d4d97a82ae6a0df341ac9, b231c0376cb149d39523ac690c896539; no voters: b382c1aff28e4d7c875ca75c180ace1e
I20260925 13:12:43.744531 7790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:12:43.744597 7789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 13:12:43.744630 7789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b382c1aff28e4d7c875ca75c180ace1e)
I20260925 13:12:43.744660 7789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:12:43.745229 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:43.745258 7789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:43.745349 7790 leader_election.cc:288] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 2 election: Requested vote from peers b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953), 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61:42371)
I20260925 13:12:43.745391 7789 leader_election.cc:288] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [CANDIDATE]: Term 2 election: Requested vote from peers b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953), b231c0376cb149d39523ac690c896539 (127.0.98.60:46699)
I20260925 13:12:43.745488 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.745411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46264)
0925 13:12:43.745440 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.745463 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 13:12:43.745546 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b231c0376cb149d39523ac690c896539" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "379d214d810d4d97a82ae6a0df341ac9"
I20260925 13:12:43.745590 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.745463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46266)
0925 13:12:43.745514 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.745569 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:12:43.745589 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "379d214d810d4d97a82ae6a0df341ac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b231c0376cb149d39523ac690c896539"
I20260925 13:12:43.745630 7097 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b231c0376cb149d39523ac690c896539 in current term 2: Already voted for candidate 379d214d810d4d97a82ae6a0df341ac9 in this term.
I20260925 13:12:43.745682 7275 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 379d214d810d4d97a82ae6a0df341ac9 in current term 2: Already voted for candidate b231c0376cb149d39523ac690c896539 in this term.
I20260925 13:12:43.746033 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.745947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46276)
0925 13:12:43.745995 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.746017 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 13:12:43.746107 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:43.746012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:46282)
0925 13:12:43.746060 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:43.746083 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:12:43.746107 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.746146 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 379d214d810d4d97a82ae6a0df341ac9; no voters: b231c0376cb149d39523ac690c896539, b382c1aff28e4d7c875ca75c180ace1e
W20260925 13:12:43.746176 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:43.746203 7789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 13:12:43.746217 7258 leader_election.cc:302] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b231c0376cb149d39523ac690c896539; no voters: 379d214d810d4d97a82ae6a0df341ac9, b382c1aff28e4d7c875ca75c180ace1e
I20260925 13:12:43.746352 7790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 13:12:44.237452 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:44.237110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:42279)
0925 13:12:44.237263 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:44.237387 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260925 13:12:44.238247 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:44.238057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:47529)
0925 13:12:44.238163 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:44.238199 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":85,"threads_started":1}
I20260925 13:12:44.238557 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:44.238265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:52101)
0925 13:12:44.238486 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:44.238525 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":56,"threads_started":1}
I20260925 13:12:45.238226 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.237975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:37673)
0925 13:12:45.238125 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.238166 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260925 13:12:45.238862 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.238668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:56081)
0925 13:12:45.238770 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.238800 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260925 13:12:45.239114 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.238963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:55443)
0925 13:12:45.239057 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.239085 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
I20260925 13:12:45.744426 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:45.744527 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:45.744800 7805 leader_election.cc:288] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953), 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61:42371)
I20260925 13:12:45.745065 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b231c0376cb149d39523ac690c896539" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "379d214d810d4d97a82ae6a0df341ac9" is_pre_election: true
I20260925 13:12:45.745172 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b231c0376cb149d39523ac690c896539 in term 2.
I20260925 13:12:45.745178 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.744956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:54926)
0925 13:12:45.745071 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.745130 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":48,"threads_started":1}
I20260925 13:12:45.745345 7259 leader_election.cc:302] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379d214d810d4d97a82ae6a0df341ac9, b231c0376cb149d39523ac690c896539; no voters:
I20260925 13:12:45.745427 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 13:12:45.745500 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:45.745544 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 2 FOLLOWER]: Advancing to term 3
I20260925 13:12:45.745577 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.745484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:54936)
0925 13:12:45.745523 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.745553 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 13:12:45.745651 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:45.746299 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:45.746433 7805 leader_election.cc:288] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 3 election: Requested vote from peers b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953), 379d214d810d4d97a82ae6a0df341ac9 (127.0.98.61:42371)
I20260925 13:12:45.746598 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.746491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:54948)
0925 13:12:45.746546 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.746583 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:12:45.746623 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b231c0376cb149d39523ac690c896539" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "379d214d810d4d97a82ae6a0df341ac9"
I20260925 13:12:45.746711 7097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 13:12:45.746933 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.746843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:54958)
0925 13:12:45.746887 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.746910 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 13:12:45.747056 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
I20260925 13:12:45.747444 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b231c0376cb149d39523ac690c896539 in term 3.
I20260925 13:12:45.747594 7259 leader_election.cc:302] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 379d214d810d4d97a82ae6a0df341ac9, b231c0376cb149d39523ac690c896539; no voters: b382c1aff28e4d7c875ca75c180ace1e
I20260925 13:12:45.747699 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 13:12:45.747818 7805 raft_consensus.cc:686] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [term 3 LEADER]: Becoming Leader. State: Replica: b231c0376cb149d39523ac690c896539, State: Running, Role: LEADER
I20260925 13:12:45.747911 7805 consensus_queue.cc:237] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } }
I20260925 13:12:45.748273 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b231c0376cb149d39523ac690c896539. Latest consensus state: current_term: 3 leader_uuid: "b231c0376cb149d39523ac690c896539" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:45.748373 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:45.748551 7810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:45.749027 7810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:45.749128 7810 catalog_manager.cc:1259] Loaded cluster ID: 7fed07700c7b40d0beffc8ed3689405b
I20260925 13:12:45.749161 7810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:45.749356 7810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:45.749451 7810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Loaded TSK: 0
I20260925 13:12:45.749598 7810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:12:45.834082 7264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:45.835717 7264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:45.836824 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.836697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:54978)
0925 13:12:45.836756 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.836793 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:12:45.836812 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [term 3 FOLLOWER]: Refusing update from remote peer b231c0376cb149d39523ac690c896539: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 13:12:45.837050 7808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 13:12:45.837168 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.837070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.0.1:54988)
0925 13:12:45.837123 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.837150 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 13:12:45.837249 7258 consensus_peers.cc:487] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 -> Peer b382c1aff28e4d7c875ca75c180ace1e (127.0.98.62:42953): Couldn't send request to peer b382c1aff28e4d7c875ca75c180ace1e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 13:12:45.837713 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b231c0376cb149d39523ac690c896539. Latest consensus state: current_term: 3 leader_uuid: "b231c0376cb149d39523ac690c896539" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:45.837822 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:12:45.838268 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b231c0376cb149d39523ac690c896539" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:45.838363 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:45.838310 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b231c0376cb149d39523ac690c896539" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b382c1aff28e4d7c875ca75c180ace1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } peers { permanent_uuid: "379d214d810d4d97a82ae6a0df341ac9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42371 } attrs { promote: false } } peers { permanent_uuid: "b231c0376cb149d39523ac690c896539" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46699 } attrs { promote: false } } }
I20260925 13:12:45.838395 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379d214d810d4d97a82ae6a0df341ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:12:45.838585 7813 mvcc.cc:205] Tried to move back new op lower bound from 7333240692065480704 to 7333240691704029184. Current Snapshot: MvccSnapshot[applied={T|T < 7333240692065480704}]
I20260925 13:12:45.843254 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.840078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35187 (local address 127.0.0.1:40864)
0925 13:12:45.840463 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.840475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:45.840518 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:45.840683 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:45.840684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:45.840685 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:45.840724 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.840725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.841504 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.841505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.842340 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.842342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.842980 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.842981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.843041 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.843046 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:45.843100 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:12:45.843153 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260925 13:12:45.843395 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.840022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.0.98.3:33745)
0925 13:12:45.840127 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:45.840129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:45.840136 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:45.840160 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:45.840161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:45.840171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:45.840404 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:45.840446 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.841201 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.842468 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.843039 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.843109 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.843119 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:45.843263 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 13:12:45.843311 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260925 13:12:45.843349 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.839950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33745 (local address 127.0.0.1:35346)
0925 13:12:45.839984 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.839996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:45.840043 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:45.840252 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:45.840252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:45.840254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:45.840338 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.840340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.841295 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.841296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.842380 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.842382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.843115 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.843116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.843184 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.843188 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:45.843243 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:12:45.843285 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":6}
I20260925 13:12:45.843649 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.840245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.0.98.2:35187)
0925 13:12:45.840348 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:45.840350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:45.840567 (+ 217us) server_negotiation.cc:407] Connection header received
0925 13:12:45.840597 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:45.840598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:45.840609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:45.840668 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:45.840798 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.841433 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.842439 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.842899 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.842972 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.842985 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:45.843412 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 13:12:45.843483 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260925 13:12:45.843995 7522 tablet_service.cc:1467] Processing CreateTablet for tablet 46a15446282a4397aba8c21dc7995e5e (DEFAULT_TABLE table=test-table [id=bbf49eb296784d92aeaf97d11c18d457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:45.844070 7658 tablet_service.cc:1467] Processing CreateTablet for tablet 46a15446282a4397aba8c21dc7995e5e (DEFAULT_TABLE table=test-table [id=bbf49eb296784d92aeaf97d11c18d457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:45.844141 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a15446282a4397aba8c21dc7995e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:45.844286 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a15446282a4397aba8c21dc7995e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:45.844779 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.840023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45195 (local address 127.0.0.1:54628)
0925 13:12:45.840207 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.840216 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:45.840265 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:45.840404 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:45.840405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:45.840407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:45.840457 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.840458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.841210 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.841210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.842729 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.842731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.843536 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.843537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.844596 (+ 1059us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.844600 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:45.844673 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 13:12:45.844712 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":7,"thread_start_us":66,"threads_started":1}
I20260925 13:12:45.844980 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.840192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.98.1:45195)
0925 13:12:45.840293 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:45.840295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:45.840303 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:45.840327 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:45.840328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:45.840337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:45.840401 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:45.840541 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.841131 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.842809 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.843412 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.843490 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.843502 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:45.844858 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0925 13:12:45.844907 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
I20260925 13:12:45.845198 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 46a15446282a4397aba8c21dc7995e5e (DEFAULT_TABLE table=test-table [id=bbf49eb296784d92aeaf97d11c18d457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:45.845304 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a15446282a4397aba8c21dc7995e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:45.845690 7822 tablet_bootstrap.cc:492] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Bootstrap starting.
I20260925 13:12:45.845871 7823 tablet_bootstrap.cc:492] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Bootstrap starting.
I20260925 13:12:45.846071 7822 tablet_bootstrap.cc:654] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:45.846220 7823 tablet_bootstrap.cc:654] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:45.846583 7822 tablet_bootstrap.cc:492] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: No bootstrap required, opened a new log
I20260925 13:12:45.846662 7822 ts_tablet_manager.cc:1397] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:45.846784 7823 tablet_bootstrap.cc:492] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: No bootstrap required, opened a new log
I20260925 13:12:45.846833 7823 ts_tablet_manager.cc:1397] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:45.846815 7822 raft_consensus.cc:348] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.846891 7822 raft_consensus.cc:374] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:45.846915 7822 raft_consensus.cc:729] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4033ae0eacbd4a34be3132ac0cf8f7e4, State: Initialized, Role: FOLLOWER
I20260925 13:12:45.846932 7823 raft_consensus.cc:348] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.846953 7822 consensus_queue.cc:260] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.846995 7823 raft_consensus.cc:374] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:45.847028 7823 raft_consensus.cc:729] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741312b211e6423597d6cb2a36643288, State: Initialized, Role: FOLLOWER
I20260925 13:12:45.847065 7826 tablet_bootstrap.cc:492] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963: Bootstrap starting.
I20260925 13:12:45.847083 7822 ts_tablet_manager.cc:1428] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:45.847098 7823 consensus_queue.cc:260] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.847239 7823 ts_tablet_manager.cc:1428] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:45.848013 7588 heartbeater.cc:503] Master 127.0.98.60:46699 was elected leader, sending a full tablet report...
I20260925 13:12:45.848034 7724 heartbeater.cc:503] Master 127.0.98.60:46699 was elected leader, sending a full tablet report...
I20260925 13:12:45.848259 7826 tablet_bootstrap.cc:654] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:45.848686 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.847414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:55749)
0925 13:12:45.848603 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.848645 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":79,"threads_started":1}
I20260925 13:12:45.848867 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.847414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:42951)
0925 13:12:45.848789 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.848824 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":66,"threads_started":1}
I20260925 13:12:45.848973 7826 tablet_bootstrap.cc:492] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963: No bootstrap required, opened a new log
I20260925 13:12:45.849022 7826 ts_tablet_manager.cc:1397] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:45.849169 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.848963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:36007)
0925 13:12:45.849106 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.849140 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 13:12:45.849187 7826 raft_consensus.cc:348] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.849272 7826 raft_consensus.cc:374] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:45.849304 7826 raft_consensus.cc:729] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffbcd807594979b348c51240d2c963, State: Initialized, Role: FOLLOWER
I20260925 13:12:45.849321 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.849064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:33235)
0925 13:12:45.849275 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.849301 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260925 13:12:45.849347 7826 consensus_queue.cc:260] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.849495 7826 ts_tablet_manager.cc:1428] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:45.849853 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.849629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:49853)
0925 13:12:45.849773 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.849809 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":100,"threads_started":1}
I20260925 13:12:45.850059 7451 heartbeater.cc:503] Master 127.0.98.60:46699 was elected leader, sending a full tablet report...
I20260925 13:12:45.850224 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.850141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:44049)
0925 13:12:45.850174 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.850199 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 13:12:45.977195 7832 raft_consensus.cc:482] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:45.977345 7832 raft_consensus.cc:504] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.977795 7832 leader_election.cc:288] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745), 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187)
I20260925 13:12:45.980937 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.978115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53165 (local address 127.0.98.3:33745)
0925 13:12:45.978167 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:45.978169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:45.978389 (+ 220us) server_negotiation.cc:407] Connection header received
0925 13:12:45.978418 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:45.978419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:45.978433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:45.978494 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:45.978709 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.979154 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.980208 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.980603 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.980656 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.980665 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:45.980834 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 13:12:45.980878 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:12:45.980942 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.978026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33745 (local address 127.0.98.1:53165)
0925 13:12:45.978244 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.978255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:45.978321 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:45.978557 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:45.978558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:45.978561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:45.978630 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.978631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.979274 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.979275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.980116 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.980118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.980686 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.980687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.980763 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.980768 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:45.980812 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:12:45.980851 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":51,"thread_start_us":132,"threads_started":1}
I20260925 13:12:45.981071 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.978134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51999 (local address 127.0.98.2:35187)
0925 13:12:45.978176 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:45.978178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:45.978226 (+ 48us) server_negotiation.cc:407] Connection header received
0925 13:12:45.978260 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:45.978261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:45.978275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:45.978331 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:45.978583 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.979133 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.980288 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.980676 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.980764 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.980776 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:45.980970 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 13:12:45.981013 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:45.981068 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.978026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35187 (local address 127.0.98.1:51999)
0925 13:12:45.978093 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.978106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:45.978171 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:45.978388 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:45.978390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:45.978393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:45.978487 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.978489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.979223 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.979224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.980199 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:45.980202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:45.980806 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:45.980807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:45.980891 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:45.980894 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:45.980961 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:12:45.980998 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 13:12:45.981485 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a15446282a4397aba8c21dc7995e5e" candidate_uuid: "3cffbcd807594979b348c51240d2c963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741312b211e6423597d6cb2a36643288" is_pre_election: true
I20260925 13:12:45.981535 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a15446282a4397aba8c21dc7995e5e" candidate_uuid: "3cffbcd807594979b348c51240d2c963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" is_pre_election: true
I20260925 13:12:45.981614 7542 raft_consensus.cc:2454] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cffbcd807594979b348c51240d2c963 in term 0.
I20260925 13:12:45.981647 7678 raft_consensus.cc:2454] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cffbcd807594979b348c51240d2c963 in term 0.
I20260925 13:12:45.981932 7338 leader_election.cc:302] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3cffbcd807594979b348c51240d2c963, 4033ae0eacbd4a34be3132ac0cf8f7e4, 741312b211e6423597d6cb2a36643288; no voters:
I20260925 13:12:45.982066 7832 raft_consensus.cc:2781] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:45.982111 7832 raft_consensus.cc:482] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:45.982126 7832 raft_consensus.cc:3037] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:45.982861 7832 raft_consensus.cc:504] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.983000 7832 leader_election.cc:288] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [CANDIDATE]: Term 1 election: Requested vote from peers 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745), 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187)
I20260925 13:12:45.983150 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a15446282a4397aba8c21dc7995e5e" candidate_uuid: "3cffbcd807594979b348c51240d2c963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4"
I20260925 13:12:45.983225 7678 raft_consensus.cc:3037] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:45.983170 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a15446282a4397aba8c21dc7995e5e" candidate_uuid: "3cffbcd807594979b348c51240d2c963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741312b211e6423597d6cb2a36643288"
I20260925 13:12:45.983271 7542 raft_consensus.cc:3037] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:45.983816 7678 raft_consensus.cc:2454] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cffbcd807594979b348c51240d2c963 in term 1.
I20260925 13:12:45.983963 7542 raft_consensus.cc:2454] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cffbcd807594979b348c51240d2c963 in term 1.
I20260925 13:12:45.983973 7341 leader_election.cc:302] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffbcd807594979b348c51240d2c963, 4033ae0eacbd4a34be3132ac0cf8f7e4; no voters:
I20260925 13:12:45.984174 7832 raft_consensus.cc:2781] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:45.984365 7832 raft_consensus.cc:686] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [term 1 LEADER]: Becoming Leader. State: Replica: 3cffbcd807594979b348c51240d2c963, State: Running, Role: LEADER
I20260925 13:12:45.984459 7832 consensus_queue.cc:237] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } }
I20260925 13:12:45.984851 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:45.984676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:57353)
0925 13:12:45.984716 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:45.984823 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 13:12:45.985162 7267 catalog_manager.cc:5495] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cffbcd807594979b348c51240d2c963 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3cffbcd807594979b348c51240d2c963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cffbcd807594979b348c51240d2c963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:46.026289 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.023998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.98.1:45195)
0925 13:12:46.024062 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:46.024064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:46.024073 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:46.024122 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:46.024123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:46.024143 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:46.024209 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:46.024326 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:46.024797 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:46.025510 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:46.025750 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:46.025808 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:46.025962 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:46.026110 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:46.026170 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:46.026211 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 13:12:46.026229 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:12:46.027416 7678 raft_consensus.cc:1261] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Refusing update from remote peer 3cffbcd807594979b348c51240d2c963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:46.027578 7542 raft_consensus.cc:1261] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Refusing update from remote peer 3cffbcd807594979b348c51240d2c963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:46.027678 7835 consensus_queue.cc:1035] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4033ae0eacbd4a34be3132ac0cf8f7e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:46.027846 7832 consensus_queue.cc:1035] T 46a15446282a4397aba8c21dc7995e5e P 3cffbcd807594979b348c51240d2c963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741312b211e6423597d6cb2a36643288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:46.028906 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.028729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:40169)
0925 13:12:46.028840 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.028874 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 13:12:46.029362 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.029135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:58605)
0925 13:12:46.029298 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.029329 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 13:12:46.029294 7839 mvcc.cc:205] Tried to move back new op lower bound from 7333240692846157824 to 7333240692672856064. Current Snapshot: MvccSnapshot[applied={T|T < 7333240692846157824}]
I20260925 13:12:46.029641 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.029509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:52349)
0925 13:12:46.029579 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.029605 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 13:12:46.030341 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.030216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:55439)
0925 13:12:46.030276 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.030311 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 13:12:46.030845 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.030741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:36069)
0925 13:12:46.030797 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.030820 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:12:46.031160 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:46.031042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:41413)
0925 13:12:46.031115 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.031139 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 13:12:46.031970 7842 mvcc.cc:205] Tried to move back new op lower bound from 7333240692846157824 to 7333240692672856064. Current Snapshot: MvccSnapshot[applied={T|T < 7333240692846157824}]
I20260925 13:12:46.032142 7268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:46.032253 7268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:46.033540 7268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b231c0376cb149d39523ac690c896539: Sending DeleteTablet for 3 replicas of tablet 46a15446282a4397aba8c21dc7995e5e
I20260925 13:12:46.033813 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a15446282a4397aba8c21dc7995e5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:46 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35346
I20260925 13:12:46.033878 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a15446282a4397aba8c21dc7995e5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:46 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40864
I20260925 13:12:46.034009 7849 tablet_replica.cc:330] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: stopping tablet replica
I20260925 13:12:46.034067 7849 raft_consensus.cc:2229] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:46.034075 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a15446282a4397aba8c21dc7995e5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:46 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54628
I20260925 13:12:46.034106 7849 raft_consensus.cc:2258] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:46.034264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 7234
I20260925 13:12:46.034322 7850 tablet_replica.cc:330] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: stopping tablet replica
I20260925 13:12:46.035284 7849 ts_tablet_manager.cc:1905] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:46.034374 7850 raft_consensus.cc:2229] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:46.035940 7850 raft_consensus.cc:2258] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:46.036303 7850 ts_tablet_manager.cc:1905] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:46.036806 7849 ts_tablet_manager.cc:1918] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:46.036883 7849 log.cc:1196] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/46a15446282a4397aba8c21dc7995e5e
I20260925 13:12:46.037129 7849 ts_tablet_manager.cc:1939] T 46a15446282a4397aba8c21dc7995e5e P 4033ae0eacbd4a34be3132ac0cf8f7e4: Deleting consensus metadata
I20260925 13:12:46.037806 7259 catalog_manager.cc:4841] TS 4033ae0eacbd4a34be3132ac0cf8f7e4 (127.0.98.3:33745): tablet 46a15446282a4397aba8c21dc7995e5e (table test-table [id=bbf49eb296784d92aeaf97d11c18d457]) successfully deleted
I20260925 13:12:46.038311 7850 ts_tablet_manager.cc:1918] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:46.038373 7850 log.cc:1196] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/46a15446282a4397aba8c21dc7995e5e
I20260925 13:12:46.038605 7850 ts_tablet_manager.cc:1939] T 46a15446282a4397aba8c21dc7995e5e P 741312b211e6423597d6cb2a36643288: Deleting consensus metadata
I20260925 13:12:46.038964 7256 catalog_manager.cc:4841] TS 741312b211e6423597d6cb2a36643288 (127.0.98.2:35187): tablet 46a15446282a4397aba8c21dc7995e5e (table test-table [id=bbf49eb296784d92aeaf97d11c18d457]) successfully deleted
W20260925 13:12:46.040967 7818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:46.040868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45195 (local address 127.0.0.1:54644)
0925 13:12:46.040906 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:46.040930 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45195: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 13:12:46.041159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 7459
W20260925 13:12:46.041141 7258 catalog_manager.cc:4584] TS 3cffbcd807594979b348c51240d2c963 (127.0.98.1:45195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46a15446282a4397aba8c21dc7995e5e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45195: connect: Connection reset by peer (error 104)
I20260925 13:12:46.047400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 7595
I20260925 13:12:46.051954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 6952
I20260925 13:12:46.056500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 7141
2026-09-25T13:12:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7056 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:46 dist-test-slave-2d0s krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:46 dist-test-slave-2d0s krb5kdc[7854](info): set up 2 sockets
Sep 25 13:12:46 dist-test-slave-2d0s krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260925 13:12:47.074056 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 13:12:47.184557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:47.262670 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:47.262902 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:47.262935 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:47.264038 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:47.264114 7867 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 13:12:47.264135 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:47.264155 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:47.264184 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:47.264207 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:47.265558 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39359
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:47.265844 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:47.266057 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:47.271044 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.271106 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:47.271071 7867 server_base.cc:1034] running on GCE node
W20260925 13:12:47.271071 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:47.271474 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:47.271740 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:47.272881 7867 hybrid_clock.cc:648] HybridClock initialized: now 1790341967272867 us; error 28 us; skew 500 ppm
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:47.275306 7867 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 13:12:47.275739 7867 webserver.cc:460] Webserver started at http://127.0.98.62:33457/ using document root <none> and password file <none>
I20260925 13:12:47.275962 7867 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:47.276027 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:47.276177 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:47.277160 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "64849755469b4710b2ff4acc02f25a54"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "a7553162deb5250fe56e3c49f84e2588"
server_key_iv: "3f6c59cbf5e2159ea005d7daf409810a"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.277519 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "64849755469b4710b2ff4acc02f25a54"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "a7553162deb5250fe56e3c49f84e2588"
server_key_iv: "3f6c59cbf5e2159ea005d7daf409810a"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.278836 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:47.279521 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.279699 7867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:47.279793 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "64849755469b4710b2ff4acc02f25a54"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "a7553162deb5250fe56e3c49f84e2588"
server_key_iv: "3f6c59cbf5e2159ea005d7daf409810a"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.279872 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:47.301018 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:47.302943 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:47.303102 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:47.307420 7867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39359
I20260925 13:12:47.307492 7944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39359 every 8 connection(s)
I20260925 13:12:47.307821 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:47.308504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 7867
I20260925 13:12:47.308544 7945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:47.308605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:47.308810 392 external_mini_cluster.cc:1428] Setting key 8d7f1b48f49f0f25cf441663d2640fa2
I20260925 13:12:47.311419 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Bootstrap starting.
I20260925 13:12:47.312005 7945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:47.312273 7945 log.cc:824] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:47.312920 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: No bootstrap required, opened a new log
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:47.315089 7945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64849755469b4710b2ff4acc02f25a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39359 } }
I20260925 13:12:47.315249 7945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:47.315315 7945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64849755469b4710b2ff4acc02f25a54, State: Initialized, Role: FOLLOWER
I20260925 13:12:47.315469 7945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64849755469b4710b2ff4acc02f25a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39359 } }
I20260925 13:12:47.315557 7945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:47.315613 7945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:47.315681 7945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:47.316392 7945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64849755469b4710b2ff4acc02f25a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39359 } }
I20260925 13:12:47.316537 7945 leader_election.cc:302] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64849755469b4710b2ff4acc02f25a54; no voters:
I20260925 13:12:47.316689 7945 leader_election.cc:288] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:47.316748 7950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:47.316887 7950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [term 1 LEADER]: Becoming Leader. State: Replica: 64849755469b4710b2ff4acc02f25a54, State: Running, Role: LEADER
I20260925 13:12:47.316972 7945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:47.317059 7950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64849755469b4710b2ff4acc02f25a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39359 } }
I20260925 13:12:47.317409 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64849755469b4710b2ff4acc02f25a54. Latest consensus state: current_term: 1 leader_uuid: "64849755469b4710b2ff4acc02f25a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64849755469b4710b2ff4acc02f25a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39359 } } }
I20260925 13:12:47.317414 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64849755469b4710b2ff4acc02f25a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64849755469b4710b2ff4acc02f25a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39359 } } }
I20260925 13:12:47.317494 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:47.317493 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54 [sys.catalog]: This master's current role is: LEADER
W20260925 13:12:47.318936 7966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:12:47.319072 7966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:12:47.319173 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:47.319344 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:47.319475 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.309326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.0.98.62:39359)
0925 13:12:47.309493 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.309504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.309534 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:12:47.310055 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.310062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.310111 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.310258 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.310852 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.311826 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.312674 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.313115 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.313228 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.315715 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.315719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.315719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.315731 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.317851 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.318318 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.318321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.318321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.318414 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.318632 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.318634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.318634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.318900 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.319049 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.319162 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.319291 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260925 13:12:47.320775 7955 catalog_manager.cc:1347] Generated new cluster ID: 5b73a97d117148138aa9440c890cd56b
I20260925 13:12:47.320822 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:47.328336 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:47.329035 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:47.332232 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Generated new TSK 0
I20260925 13:12:47.332389 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 13:12:47.357016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39359
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:47.436792 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:47.437018 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:47.437055 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:47.438225 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:47.438288 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:47.438323 7971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:47.438362 7971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:47.439814 7971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:47.440068 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:47.440281 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:47.445931 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.445991 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.445948 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:47.446149 7971 server_base.cc:1034] running on GCE node
I20260925 13:12:47.446413 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:47.446697 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:47.447857 7971 hybrid_clock.cc:648] HybridClock initialized: now 1790341967447848 us; error 26 us; skew 500 ppm
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:47.450275 7971 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:12:47.450711 7971 webserver.cc:460] Webserver started at http://127.0.98.1:36805/ using document root <none> and password file <none>
I20260925 13:12:47.450942 7971 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:47.450986 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:47.451146 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:47.452242 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "761fca4a37bc434b95d03902d02f4419"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "2cb50ae761ac54c0ad34bdf30abff8fb"
server_key_iv: "f25edc10f37a81c91de282bbd5cef93b"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.452594 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "761fca4a37bc434b95d03902d02f4419"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "2cb50ae761ac54c0ad34bdf30abff8fb"
server_key_iv: "f25edc10f37a81c91de282bbd5cef93b"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.453955 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:47.454677 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.454859 7971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:47.454944 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "761fca4a37bc434b95d03902d02f4419"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "2cb50ae761ac54c0ad34bdf30abff8fb"
server_key_iv: "f25edc10f37a81c91de282bbd5cef93b"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.455015 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:47.463168 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:47.464824 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:47.464980 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:47.465445 7971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:47.465484 7971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.465505 7971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:47.465551 7971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.472399 7971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41101
I20260925 13:12:47.472460 8101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41101 every 8 connection(s)
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:47.472841 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:12:47.475492 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.465920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58863 (local address 127.0.98.62:39359)
0925 13:12:47.466007 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.466011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.466162 (+ 151us) server_negotiation.cc:407] Connection header received
0925 13:12:47.466627 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.466629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.466643 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.466719 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.467496 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.468334 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.469958 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.470359 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.470443 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.473083 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.473084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.473085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.473093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.474425 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.474774 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.474775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.474775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.474823 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.474988 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.474989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.474989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.475085 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.475139 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.475367 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.475429 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:12:47.475677 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.465877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.98.1:58863)
0925 13:12:47.466033 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.466070 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.466529 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.466780 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.466784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.467112 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:47.467380 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.467388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.468439 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.468440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.469859 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.469862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.470462 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.470463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.470605 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.471132 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:47.471142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:47.472964 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:47.474495 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.474501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.474502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.474711 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.474862 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.474863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.474863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.474941 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.475153 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:47.475158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:47.475263 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.475342 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.475454 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 13:12:47.476203 8102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39359
I20260925 13:12:47.476279 8102 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:47.476446 8102 heartbeater.cc:511] Master 127.0.98.62:39359 requested a full tablet report, sending...
I20260925 13:12:47.477080 7898 ts_manager.cc:194] Registered new tserver with Master: 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101)
I20260925 13:12:47.477979 7898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58863
I20260925 13:12:47.481822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 7971
I20260925 13:12:47.481950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:47.482199 392 external_mini_cluster.cc:1428] Setting key 069f20cd4b867eea871e97d92095d2d1
I20260925 13:12:47.483047 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.477310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.98.1:58827)
0925 13:12:47.477480 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.477490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.477543 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.477752 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.477754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.477952 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:47.477998 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.477999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.478863 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.478864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.479363 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.479365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.479685 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.479686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.479732 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.480064 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:47.480068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:47.480317 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:47.481949 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.481950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.481951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.482161 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.482389 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.482390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.482391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.482453 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.482737 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:47.482738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:47.482792 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.482846 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.482918 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260925 13:12:47.483044 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.477401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58827 (local address 127.0.98.62:39359)
0925 13:12:47.477439 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.477440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.477533 (+ 93us) server_negotiation.cc:407] Connection header received
0925 13:12:47.477628 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.477629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.477638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.477726 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.478075 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.478762 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.479434 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.479604 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.479674 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.480386 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.480387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.480387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.480393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.481835 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.482244 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.482245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.482245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.482304 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.482530 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.482532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.482532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.482675 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.482738 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.482885 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.482960 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 13:12:47.506745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39359
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:47.590297 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:47.590544 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:47.590582 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:47.591748 8111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:47.591815 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:47.591847 8111 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:47.591889 8111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:47.593364 8111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:47.593638 8111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:47.593881 8111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:47.598801 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.598834 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.598796 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:47.599083 8111 server_base.cc:1034] running on GCE node
I20260925 13:12:47.599267 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:47.599512 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:47.600703 8111 hybrid_clock.cc:648] HybridClock initialized: now 1790341967600688 us; error 29 us; skew 500 ppm
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:47.603295 8111 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:12:47.603703 8111 webserver.cc:460] Webserver started at http://127.0.98.2:38341/ using document root <none> and password file <none>
I20260925 13:12:47.603875 8111 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:47.603912 8111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:47.603998 8111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:47.604981 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "8584763e51c2480582cb404ce2dc63c5"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "8c5f3815e1dacd088cc896b791f7cffe"
server_key_iv: "fb3792d0f92da016aa06ce9266b63996"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.605300 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8584763e51c2480582cb404ce2dc63c5"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "8c5f3815e1dacd088cc896b791f7cffe"
server_key_iv: "fb3792d0f92da016aa06ce9266b63996"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.606727 8111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:47.607455 8126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.607638 8111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:47.607729 8111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "8584763e51c2480582cb404ce2dc63c5"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "8c5f3815e1dacd088cc896b791f7cffe"
server_key_iv: "fb3792d0f92da016aa06ce9266b63996"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.607782 8111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:47.623881 8111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:47.625967 8111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:47.626085 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:47.626542 8111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:47.626576 8111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.626598 8111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:47.626639 8111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.633356 8111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35053
I20260925 13:12:47.633478 8241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35053 every 8 connection(s)
I20260925 13:12:47.633814 8111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:47.637454 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.627191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54117 (local address 127.0.98.62:39359)
0925 13:12:47.627258 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.627261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.627495 (+ 234us) server_negotiation.cc:407] Connection header received
0925 13:12:47.628146 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.628148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.628162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.628241 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.629125 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.630001 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.631635 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.631985 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.632080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.634829 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.634831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.634832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.634839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.636007 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.636427 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.636428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.636429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.636485 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.636786 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.636786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.636787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.636926 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.636992 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.637281 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.637364 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:12:47.637637 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.627153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.98.2:54117)
0925 13:12:47.627355 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.627398 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.628000 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.628331 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.628336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.628717 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:47.629010 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.629017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.630121 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.630122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.631529 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.631532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.632095 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.632096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.632282 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.632907 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:47.632926 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:47.634729 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:47.636116 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.636122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.636124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.636337 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.636576 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.636577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.636577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.636689 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.636990 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:47.636997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:47.637143 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.637249 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.637407 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260925 13:12:47.638217 8242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39359
I20260925 13:12:47.638321 8242 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:47.638496 8242 heartbeater.cc:511] Master 127.0.98.62:39359 requested a full tablet report, sending...
I20260925 13:12:47.638957 7899 ts_manager.cc:194] Registered new tserver with Master: 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053)
I20260925 13:12:47.639523 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54117
I20260925 13:12:47.641515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 8111
I20260925 13:12:47.641650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:47.641913 392 external_mini_cluster.cc:1428] Setting key a675123fcbf0e722a6e2bc9dbbdde5d4
I20260925 13:12:47.645165 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.639378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60829 (local address 127.0.98.62:39359)
0925 13:12:47.639418 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.639419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.639478 (+ 59us) server_negotiation.cc:407] Connection header received
0925 13:12:47.639562 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.639563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.639571 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.639648 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.640006 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.640611 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.641255 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.641436 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.641508 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.642582 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.642583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.642583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.642589 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.643993 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.644363 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.644364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.644365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.644422 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.644622 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.644623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.644624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.644776 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.644849 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.645003 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.645075 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:47.645190 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.639280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.98.2:60829)
0925 13:12:47.639420 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.639431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.639494 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.639654 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.639655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.639862 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:47.639934 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.639935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.640701 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.640702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.641186 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.641187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.641510 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.641511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.641582 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.642121 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:47.642127 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:47.642502 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:47.644099 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.644100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.644108 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.644276 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.644488 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.644489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.644489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.644547 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.644855 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:47.644856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:47.644898 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.644979 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.645076 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 13:12:47.668533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39359
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:47.749347 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:47.749553 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:47.749590 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:47.750662 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:47.750698 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:47.750718 8251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:12:47.750734 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:47.752151 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:47.752342 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:47.752517 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:12:47.757462 8251 server_base.cc:1034] running on GCE node
W20260925 13:12:47.757468 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.757570 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:47.757474 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:47.757898 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:47.758195 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:47.759343 8251 hybrid_clock.cc:648] HybridClock initialized: now 1790341967759332 us; error 31 us; skew 500 ppm
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:47.762017 8251 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:12:47.762457 8251 webserver.cc:460] Webserver started at http://127.0.98.3:34557/ using document root <none> and password file <none>
I20260925 13:12:47.762696 8251 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:47.762766 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:47.762903 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:47.763926 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "35cfde93644544ad9195cc2a1f847b62"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "079ae8897eb26529f5a7fc225eef60e9"
server_key_iv: "f2534ceafebba106cd21a15bf92e9916"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.764222 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "35cfde93644544ad9195cc2a1f847b62"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "079ae8897eb26529f5a7fc225eef60e9"
server_key_iv: "f2534ceafebba106cd21a15bf92e9916"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.765475 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:47.766326 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.766521 8251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:47.766644 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "35cfde93644544ad9195cc2a1f847b62"
format_stamp: "Formatted at 2026-09-25 13:12:47 on dist-test-slave-2d0s"
server_key: "079ae8897eb26529f5a7fc225eef60e9"
server_key_iv: "f2534ceafebba106cd21a15bf92e9916"
server_key_version: "encryptionkey@0"
I20260925 13:12:47.766722 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:47.784168 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:47.786592 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:47.786767 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:47.787272 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:47.787338 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.787386 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:47.787412 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:47.794041 8251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41005
I20260925 13:12:47.794085 8381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41005 every 8 connection(s)
I20260925 13:12:47.794410 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:47.797763 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.787766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43849 (local address 127.0.98.62:39359)
0925 13:12:47.787823 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.787825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.788033 (+ 208us) server_negotiation.cc:407] Connection header received
0925 13:12:47.788602 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.788603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.788615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.788672 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.789533 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.790398 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.791987 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.792314 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.792406 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.794966 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.794968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.794968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.794975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.796389 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.796797 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.796797 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.796798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.796846 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.797110 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.797110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.797111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.797208 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.797264 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.797551 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.797624 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:12:47.797899 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.787759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.98.3:43849)
0925 13:12:47.787911 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.787945 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.788485 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.788745 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.788749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.789122 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:47.789413 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.789420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.790496 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.790497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.791866 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.791868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.792406 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.792407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.792551 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.793133 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:47.793152 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:47.794883 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:47.796476 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.796481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.796483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.796720 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.796927 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.796928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.796928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.797046 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.797266 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:47.797271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:47.797414 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.797498 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.797617 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 13:12:47.798406 8382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39359
I20260925 13:12:47.798496 8382 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:47.798681 8382 heartbeater.cc:511] Master 127.0.98.62:39359 requested a full tablet report, sending...
I20260925 13:12:47.799472 7898 ts_manager.cc:194] Registered new tserver with Master: 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:47.799850 7898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43849
I20260925 13:12:47.803541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 8251
I20260925 13:12:47.803685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:47.803954 392 external_mini_cluster.cc:1428] Setting key 2db0c2a354984f03df8dd60874c54ac3
I20260925 13:12:47.804651 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.799449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.98.3:34313)
0925 13:12:47.799571 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.799582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.799690 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.799801 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.799801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.799971 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:47.800044 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.800046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.800659 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.800660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.801158 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.801159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.801524 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.801524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.801587 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.802008 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:47.802013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:47.802268 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:47.803511 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.803512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.803513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.803728 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.803952 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:47.803953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:47.803953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:47.804014 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:47.804325 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:47.804326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:47.804372 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.804447 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.804546 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260925 13:12:47.804663 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.799514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34313 (local address 127.0.98.62:39359)
0925 13:12:47.799556 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.799558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.799675 (+ 117us) server_negotiation.cc:407] Connection header received
0925 13:12:47.799736 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.799737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.799747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.799789 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.800107 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.800563 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.801251 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.801444 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.801523 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.802352 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.802354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.802354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.802361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.803416 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.803790 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.803791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.803792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.803858 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.804103 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.804104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.804105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.804253 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.804326 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.804485 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.804563 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:47.804889 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 13:12:47.826012 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:47.820518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.0.98.62:39359)
0925 13:12:47.820574 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.820576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.820586 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:47.820656 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.820657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.820674 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.820737 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.821295 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.821874 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.822627 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.822821 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.822906 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.825915 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33002: BlockingRecv error: recv got EOF from 127.0.0.1:33002 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 13:12:47 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:47.833076 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.826729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.0.98.62:39359)
0925 13:12:47.826774 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.826775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.826868 (+ 93us) server_negotiation.cc:407] Connection header received
0925 13:12:47.826903 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.826904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.826912 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.826958 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:47.827373 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.827826 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.828475 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.828672 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.828724 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.830880 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:47.830881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:47.830882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:47.830887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:47.832104 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.832458 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.832458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.832459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.832506 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:47.832711 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:47.832712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:47.832712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:47.832804 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:47.832875 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.832957 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.833010 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:12:47.834728 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:47.836230 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:47.842890 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.838520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.0.1:51642)
0925 13:12:47.838829 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.838870 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.838945 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.839290 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.839291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.839296 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:47.839404 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.839407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.840311 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.840312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.841512 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.841514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.842562 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.842563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.842649 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.842665 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.842726 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.842793 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":67,"threads_started":1}
I20260925 13:12:47.843268 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.838866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41005 (local address 127.0.0.1:52204)
0925 13:12:47.839125 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.839134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.839179 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.839432 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.839433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.839435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:47.839475 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.839476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.840695 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.840696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.842029 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.842031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.843028 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.843029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.843097 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.843101 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.843156 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.843197 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":58,"threads_started":1}
I20260925 13:12:47.843308 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.838970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.0.98.3:41005)
0925 13:12:47.839121 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.839125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.839240 (+ 115us) server_negotiation.cc:407] Connection header received
0925 13:12:47.839279 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.839281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.839296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.839366 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:47.839553 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.840607 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.842121 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.842842 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.842904 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.842988 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.843170 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.843211 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260925 13:12:47.843348 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.838624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.0.98.2:35053)
0925 13:12:47.838775 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.838780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.839120 (+ 340us) server_negotiation.cc:407] Connection header received
0925 13:12:47.839169 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.839172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.839190 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.839480 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:47.839546 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.840223 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.841595 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.842453 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.843076 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.843169 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.843196 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.843244 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260925 13:12:47.843582 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.838548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.0.1:55028)
0925 13:12:47.838973 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.838983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.839044 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.839286 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.839290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.839292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:47.839398 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.839400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.840728 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.840729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.841664 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.841667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.842916 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.842918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.843140 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.843143 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.843212 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.843251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":85,"threads_started":1}
I20260925 13:12:47.843904 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.838657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.0.98.1:41101)
0925 13:12:47.838826 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.838831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.839055 (+ 224us) server_negotiation.cc:407] Connection header received
0925 13:12:47.839106 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.839108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.839126 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.839201 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:47.839480 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.840607 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.841807 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.842827 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.842906 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.843084 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.843312 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.843825 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260925 13:12:47.844022 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6152a48ee4468bb0a08d7df5a0c08d (DEFAULT_TABLE table=test-table [id=c12c22ca264549cb949962dd35e5bcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:47.844411 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6152a48ee4468bb0a08d7df5a0c08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:47.844518 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6152a48ee4468bb0a08d7df5a0c08d (DEFAULT_TABLE table=test-table [id=c12c22ca264549cb949962dd35e5bcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:47.844851 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6152a48ee4468bb0a08d7df5a0c08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:47.847592 8410 tablet_bootstrap.cc:492] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Bootstrap starting.
I20260925 13:12:47.847925 8411 tablet_bootstrap.cc:492] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Bootstrap starting.
I20260925 13:12:47.848270 8410 tablet_bootstrap.cc:654] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:47.848199 8316 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6152a48ee4468bb0a08d7df5a0c08d (DEFAULT_TABLE table=test-table [id=c12c22ca264549cb949962dd35e5bcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:47.848506 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6152a48ee4468bb0a08d7df5a0c08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:47.848539 8411 tablet_bootstrap.cc:654] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:47.848610 8410 log.cc:824] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:47.848863 8411 log.cc:824] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:47.849325 8410 tablet_bootstrap.cc:492] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: No bootstrap required, opened a new log
I20260925 13:12:47.849380 8410 ts_tablet_manager.cc:1397] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:47.850793 8410 raft_consensus.cc:348] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.850917 8410 raft_consensus.cc:374] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:47.850939 8410 raft_consensus.cc:729] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Initialized, Role: FOLLOWER
I20260925 13:12:47.851055 8410 consensus_queue.cc:260] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.851322 8410 ts_tablet_manager.cc:1428] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:12:47.851473 8242 heartbeater.cc:503] Master 127.0.98.62:39359 was elected leader, sending a full tablet report...
I20260925 13:12:47.851930 8415 tablet_bootstrap.cc:492] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Bootstrap starting.
I20260925 13:12:47.852130 8411 tablet_bootstrap.cc:492] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: No bootstrap required, opened a new log
I20260925 13:12:47.852208 8411 ts_tablet_manager.cc:1397] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 13:12:47.852484 8415 tablet_bootstrap.cc:654] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:47.852777 8415 log.cc:824] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:47.853453 8415 tablet_bootstrap.cc:492] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: No bootstrap required, opened a new log
I20260925 13:12:47.853533 8415 ts_tablet_manager.cc:1397] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:12:47.854061 8411 raft_consensus.cc:348] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.854182 8411 raft_consensus.cc:374] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:47.854202 8411 raft_consensus.cc:729] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Initialized, Role: FOLLOWER
I20260925 13:12:47.854321 8411 consensus_queue.cc:260] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.854598 8411 ts_tablet_manager.cc:1428] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:12:47.854635 8102 heartbeater.cc:503] Master 127.0.98.62:39359 was elected leader, sending a full tablet report...
I20260925 13:12:47.855206 8415 raft_consensus.cc:348] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.855356 8415 raft_consensus.cc:374] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:47.855404 8415 raft_consensus.cc:729] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cfde93644544ad9195cc2a1f847b62, State: Initialized, Role: FOLLOWER
I20260925 13:12:47.855512 8415 consensus_queue.cc:260] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.855758 8415 ts_tablet_manager.cc:1428] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:12:47.855816 8382 heartbeater.cc:503] Master 127.0.98.62:39359 was elected leader, sending a full tablet report...
W20260925 13:12:47.884553 8243 tablet.cc:2369] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:47.955971 8417 raft_consensus.cc:482] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:47.956089 8417 raft_consensus.cc:504] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.956442 8417 leader_election.cc:288] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:47.960278 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.956726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40711 (local address 127.0.98.3:41005)
0925 13:12:47.956777 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.956779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.956929 (+ 150us) server_negotiation.cc:407] Connection header received
0925 13:12:47.956957 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.956958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.956970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.957043 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:47.957217 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.957752 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.959297 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.959865 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.959948 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.959957 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.960170 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.960207 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:12:47.960278 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.956703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57655 (local address 127.0.98.2:35053)
0925 13:12:47.956746 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.956749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.956758 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:47.956830 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.956832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.956847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.956919 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:47.957149 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.957827 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.959297 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.959865 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.959948 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.959958 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.960170 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.960207 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:12:47.960295 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.956615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.98.1:57655)
0925 13:12:47.956692 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.956704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.956748 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.956955 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.956956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.956958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:47.957055 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.957057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.957912 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.957913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.959087 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.959090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.959972 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.959973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.960068 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.960073 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.960147 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.960191 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":31}
I20260925 13:12:47.960314 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.956624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41005 (local address 127.0.98.1:40711)
0925 13:12:47.956824 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:47.956833 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:47.956907 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:47.957047 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:47.957048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:47.957049 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:47.957143 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.957144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.957822 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.957823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.959087 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:47.959090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.959972 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.959973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:47.960068 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.960073 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:47.960140 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:12:47.960191 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":76,"thread_start_us":77,"threads_started":1}
I20260925 13:12:47.960739 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6152a48ee4468bb0a08d7df5a0c08d" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8584763e51c2480582cb404ce2dc63c5" is_pre_election: true
I20260925 13:12:47.960908 8196 raft_consensus.cc:2454] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 0.
I20260925 13:12:47.960942 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6152a48ee4468bb0a08d7df5a0c08d" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62" is_pre_election: true
I20260925 13:12:47.961069 8336 raft_consensus.cc:2454] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 0.
I20260925 13:12:47.961141 7988 leader_election.cc:302] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761fca4a37bc434b95d03902d02f4419, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:47.961302 8417 raft_consensus.cc:2781] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:47.961361 8417 raft_consensus.cc:482] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:47.961393 8417 raft_consensus.cc:3037] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:47.962169 8417 raft_consensus.cc:504] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.962316 8417 leader_election.cc:288] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 election: Requested vote from peers 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:47.962469 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6152a48ee4468bb0a08d7df5a0c08d" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8584763e51c2480582cb404ce2dc63c5"
I20260925 13:12:47.962509 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6152a48ee4468bb0a08d7df5a0c08d" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62"
I20260925 13:12:47.962553 8196 raft_consensus.cc:3037] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:47.962569 8336 raft_consensus.cc:3037] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:47.963276 8336 raft_consensus.cc:2454] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 1.
I20260925 13:12:47.963307 8196 raft_consensus.cc:2454] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 1.
I20260925 13:12:47.963781 7990 leader_election.cc:302] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cfde93644544ad9195cc2a1f847b62, 761fca4a37bc434b95d03902d02f4419; no voters:
I20260925 13:12:47.963904 8417 raft_consensus.cc:2781] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:47.964042 8417 raft_consensus.cc:686] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 1 LEADER]: Becoming Leader. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Running, Role: LEADER
I20260925 13:12:47.964156 8417 consensus_queue.cc:237] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:47.964967 7899 catalog_manager.cc:5495] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 reported cstate change: term changed from 0 to 1, leader changed from <none> to 761fca4a37bc434b95d03902d02f4419 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "761fca4a37bc434b95d03902d02f4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } health_report { overall_health: UNKNOWN } } }
W20260925 13:12:47.973631 8103 tablet.cc:2369] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:47.988976 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:47.986517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.0.98.1:41101)
0925 13:12:47.986587 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:47.986589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:47.986630 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:12:47.986663 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:47.986664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:47.986684 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:47.986760 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:47.986921 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.987387 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.988170 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:47.988367 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:47.988457 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:47.988630 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:47.988739 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:47.988834 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:47.988858 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 13:12:47.988879 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:12:47.990829 8336 raft_consensus.cc:1261] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Refusing update from remote peer 761fca4a37bc434b95d03902d02f4419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:47.990830 8196 raft_consensus.cc:1261] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Refusing update from remote peer 761fca4a37bc434b95d03902d02f4419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:47.991168 8417 consensus_queue.cc:1035] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:47.991325 8420 consensus_queue.cc:1035] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:47.993770 8423 mvcc.cc:205] Tried to move back new op lower bound from 7333240700887805952 to 7333240700781887488. Current Snapshot: MvccSnapshot[applied={T|T < 7333240700887805952}]
I20260925 13:12:47.993947 8425 mvcc.cc:205] Tried to move back new op lower bound from 7333240700887805952 to 7333240700781887488. Current Snapshot: MvccSnapshot[applied={T|T < 7333240700887805952}]
I20260925 13:12:47.996297 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:47.996395 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:47.997144 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Sending DeleteTablet for 3 replicas of tablet 0d6152a48ee4468bb0a08d7df5a0c08d
I20260925 13:12:47.997467 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6152a48ee4468bb0a08d7df5a0c08d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55028
I20260925 13:12:47.997593 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6152a48ee4468bb0a08d7df5a0c08d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51642
I20260925 13:12:47.997731 8432 tablet_replica.cc:330] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: stopping tablet replica
I20260925 13:12:47.997848 8432 raft_consensus.cc:2229] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:47.997944 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6152a48ee4468bb0a08d7df5a0c08d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52204
I20260925 13:12:47.997982 8432 raft_consensus.cc:2258] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:47.998188 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:47.998317 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:47.998427 8432 ts_tablet_manager.cc:1905] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:47.999572 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 7f792512703244afaf3aede0737f5933 (DEFAULT_TABLE table=test-table [id=1af16532c4534b03bd377060e1af4e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:47.999632 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 7f792512703244afaf3aede0737f5933 (DEFAULT_TABLE table=test-table [id=1af16532c4534b03bd377060e1af4e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:47.999652 8316 tablet_service.cc:1467] Processing CreateTablet for tablet 7f792512703244afaf3aede0737f5933 (DEFAULT_TABLE table=test-table [id=1af16532c4534b03bd377060e1af4e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:47.999687 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f792512703244afaf3aede0737f5933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:47.999752 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f792512703244afaf3aede0737f5933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:47.999751 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f792512703244afaf3aede0737f5933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.000990 8415 tablet_bootstrap.cc:492] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Bootstrap starting.
I20260925 13:12:48.001005 8411 tablet_bootstrap.cc:492] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Bootstrap starting.
I20260925 13:12:48.001205 8410 tablet_bootstrap.cc:492] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Bootstrap starting.
I20260925 13:12:48.001415 8411 tablet_bootstrap.cc:654] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.001624 8410 tablet_bootstrap.cc:654] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.001765 8415 tablet_bootstrap.cc:654] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.002092 8411 tablet_bootstrap.cc:492] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: No bootstrap required, opened a new log
I20260925 13:12:48.002156 8411 ts_tablet_manager.cc:1397] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:48.002238 8410 tablet_bootstrap.cc:492] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: No bootstrap required, opened a new log
I20260925 13:12:48.002291 8410 ts_tablet_manager.cc:1397] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:48.002295 8411 raft_consensus.cc:348] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.002369 8411 raft_consensus.cc:374] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.002396 8411 raft_consensus.cc:729] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.002435 8411 consensus_queue.cc:260] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.002452 8410 raft_consensus.cc:348] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.002519 8411 ts_tablet_manager.cc:1428] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.002524 8410 raft_consensus.cc:374] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.002548 8410 raft_consensus.cc:729] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.002595 8410 consensus_queue.cc:260] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.002717 8410 ts_tablet_manager.cc:1428] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.003683 8432 ts_tablet_manager.cc:1918] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:48.003773 8432 log.cc:1196] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/0d6152a48ee4468bb0a08d7df5a0c08d
I20260925 13:12:48.003994 8432 ts_tablet_manager.cc:1939] T 0d6152a48ee4468bb0a08d7df5a0c08d P 761fca4a37bc434b95d03902d02f4419: Deleting consensus metadata
I20260925 13:12:48.004218 8434 tablet_replica.cc:330] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: stopping tablet replica
I20260925 13:12:48.004326 8434 raft_consensus.cc:2229] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.004475 8434 raft_consensus.cc:2258] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.004889 7885 catalog_manager.cc:4841] TS 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101): tablet 0d6152a48ee4468bb0a08d7df5a0c08d (table test-table [id=c12c22ca264549cb949962dd35e5bcaa]) successfully deleted
I20260925 13:12:48.005138 8434 ts_tablet_manager.cc:1905] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.006381 8434 ts_tablet_manager.cc:1918] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:48.006464 8434 log.cc:1196] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/0d6152a48ee4468bb0a08d7df5a0c08d
I20260925 13:12:48.006685 8434 ts_tablet_manager.cc:1939] T 0d6152a48ee4468bb0a08d7df5a0c08d P 35cfde93644544ad9195cc2a1f847b62: Deleting consensus metadata
I20260925 13:12:48.007072 7886 catalog_manager.cc:4841] TS 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005): tablet 0d6152a48ee4468bb0a08d7df5a0c08d (table test-table [id=c12c22ca264549cb949962dd35e5bcaa]) successfully deleted
I20260925 13:12:48.009781 8433 tablet_replica.cc:330] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: stopping tablet replica
I20260925 13:12:48.009912 8433 raft_consensus.cc:2229] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.010054 8433 raft_consensus.cc:2258] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.010485 8433 ts_tablet_manager.cc:1905] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.010528 8415 tablet_bootstrap.cc:492] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: No bootstrap required, opened a new log
I20260925 13:12:48.010571 8415 ts_tablet_manager.cc:1397] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260925 13:12:48.010730 8415 raft_consensus.cc:348] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.010805 8415 raft_consensus.cc:374] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.010824 8415 raft_consensus.cc:729] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cfde93644544ad9195cc2a1f847b62, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.010865 8415 consensus_queue.cc:260] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.010947 8415 ts_tablet_manager.cc:1428] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.012224 8433 ts_tablet_manager.cc:1918] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:48.012311 8433 log.cc:1196] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/0d6152a48ee4468bb0a08d7df5a0c08d
I20260925 13:12:48.012535 8433 ts_tablet_manager.cc:1939] T 0d6152a48ee4468bb0a08d7df5a0c08d P 8584763e51c2480582cb404ce2dc63c5: Deleting consensus metadata
I20260925 13:12:48.012959 7884 catalog_manager.cc:4841] TS 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053): tablet 0d6152a48ee4468bb0a08d7df5a0c08d (table test-table [id=c12c22ca264549cb949962dd35e5bcaa]) successfully deleted
W20260925 13:12:48.044997 8383 tablet.cc:2369] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:12:48.134451 8418 raft_consensus.cc:482] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:48.134599 8418 raft_consensus.cc:504] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.134994 8418 leader_election.cc:288] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101), 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053)
I20260925 13:12:48.137871 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.135187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.98.3:42429)
0925 13:12:48.135243 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.135256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.135305 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.135486 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.135487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.135488 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:48.135559 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.135561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.136257 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.136258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.137046 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.137048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.137601 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.137602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.137691 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.137695 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.137743 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.137784 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":33}
I20260925 13:12:48.137862 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.135290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42429 (local address 127.0.98.2:35053)
0925 13:12:48.135368 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.135370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.135379 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:48.135411 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.135412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.135419 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.135483 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:48.135667 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.136168 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.137163 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.137519 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.137635 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.137645 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.137758 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.137795 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 13:12:48.138100 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.135201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.98.3:60211)
0925 13:12:48.135346 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.135356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.135420 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.135559 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.135560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.135561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:48.135671 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.135672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.136310 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.136311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.137070 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.137072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.137624 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.137625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.137725 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.137728 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.137807 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.138002 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":14,"thread_start_us":59,"threads_started":1}
I20260925 13:12:48.138222 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.135303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60211 (local address 127.0.98.1:41101)
0925 13:12:48.135354 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.135357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.135407 (+ 50us) server_negotiation.cc:407] Connection header received
0925 13:12:48.135474 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.135475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.135484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.135555 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:48.135762 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.136225 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.137169 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.137535 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.137618 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.137629 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.137811 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.137849 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:12:48.138222 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f792512703244afaf3aede0737f5933" candidate_uuid: "35cfde93644544ad9195cc2a1f847b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8584763e51c2480582cb404ce2dc63c5" is_pre_election: true
I20260925 13:12:48.138321 8196 raft_consensus.cc:2454] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cfde93644544ad9195cc2a1f847b62 in term 0.
I20260925 13:12:48.138412 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f792512703244afaf3aede0737f5933" candidate_uuid: "35cfde93644544ad9195cc2a1f847b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761fca4a37bc434b95d03902d02f4419" is_pre_election: true
I20260925 13:12:48.138531 8056 raft_consensus.cc:2454] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cfde93644544ad9195cc2a1f847b62 in term 0.
I20260925 13:12:48.138553 8268 leader_election.cc:302] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cfde93644544ad9195cc2a1f847b62, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.138679 8418 raft_consensus.cc:2781] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:48.138718 8418 raft_consensus.cc:482] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:48.138737 8418 raft_consensus.cc:3037] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.139503 8418 raft_consensus.cc:504] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.139642 8418 leader_election.cc:288] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [CANDIDATE]: Term 1 election: Requested vote from peers 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101), 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053)
I20260925 13:12:48.139803 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f792512703244afaf3aede0737f5933" candidate_uuid: "35cfde93644544ad9195cc2a1f847b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761fca4a37bc434b95d03902d02f4419"
I20260925 13:12:48.139829 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f792512703244afaf3aede0737f5933" candidate_uuid: "35cfde93644544ad9195cc2a1f847b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8584763e51c2480582cb404ce2dc63c5"
I20260925 13:12:48.139892 8196 raft_consensus.cc:3037] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.139897 8056 raft_consensus.cc:3037] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.140457 8196 raft_consensus.cc:2454] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cfde93644544ad9195cc2a1f847b62 in term 1.
I20260925 13:12:48.140570 8056 raft_consensus.cc:2454] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cfde93644544ad9195cc2a1f847b62 in term 1.
I20260925 13:12:48.140611 8268 leader_election.cc:302] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cfde93644544ad9195cc2a1f847b62, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.140693 8418 raft_consensus.cc:2781] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:48.140805 8418 raft_consensus.cc:686] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 1 LEADER]: Becoming Leader. State: Replica: 35cfde93644544ad9195cc2a1f847b62, State: Running, Role: LEADER
I20260925 13:12:48.140877 8418 consensus_queue.cc:237] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.141736 7897 catalog_manager.cc:5495] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "35cfde93644544ad9195cc2a1f847b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } health_report { overall_health: HEALTHY } } }
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:48.185317 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.178350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39359 (local address 127.0.0.1:33022)
0925 13:12:48.178556 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.178567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.178656 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.178822 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.178822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.179152 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:48.179199 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.179201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.179860 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.179862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.180386 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.180388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.180727 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.180728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.180776 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.181279 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:48.181283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:48.182702 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:48.184380 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:48.184386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:48.184388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:48.184579 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:48.184779 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:48.184780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:48.184780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:48.184836 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:48.185035 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:48.185040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:48.185082 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.185132 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.185226 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
I20260925 13:12:48.185366 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.178446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.0.98.62:39359)
0925 13:12:48.178526 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.178528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.178635 (+ 107us) server_negotiation.cc:407] Connection header received
0925 13:12:48.178733 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.178734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.178745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.178817 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:48.179291 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.179782 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.180473 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.180649 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.180727 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.182776 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:48.182777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:48.182777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:48.182784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:48.184307 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:48.184674 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:48.184675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:48.184675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:48.184723 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:48.184890 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:48.184890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:48.184890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:48.184987 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:48.185034 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.185151 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.185274 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":52}
I20260925 13:12:48.187366 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 13:12:48.187582 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:48.188932 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 489065f7fff04e63aaffa0176effd0fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98b7a46abaf24ea8ab23e8d4d056c717]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:48.188944 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 489065f7fff04e63aaffa0176effd0fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98b7a46abaf24ea8ab23e8d4d056c717]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:48.188944 8316 tablet_service.cc:1467] Processing CreateTablet for tablet 489065f7fff04e63aaffa0176effd0fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98b7a46abaf24ea8ab23e8d4d056c717]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:12:48.189087 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489065f7fff04e63aaffa0176effd0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.189087 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489065f7fff04e63aaffa0176effd0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.189103 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489065f7fff04e63aaffa0176effd0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.190379 8411 tablet_bootstrap.cc:492] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: Bootstrap starting.
I20260925 13:12:48.190608 8415 tablet_bootstrap.cc:492] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: Bootstrap starting.
I20260925 13:12:48.190744 8411 tablet_bootstrap.cc:654] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.191011 8415 tablet_bootstrap.cc:654] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.191287 8411 tablet_bootstrap.cc:492] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: No bootstrap required, opened a new log
I20260925 13:12:48.191329 8411 ts_tablet_manager.cc:1397] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:48.191375 8410 tablet_bootstrap.cc:492] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: Bootstrap starting.
I20260925 13:12:48.191447 8411 raft_consensus.cc:348] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.191519 8411 raft_consensus.cc:374] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.191545 8411 raft_consensus.cc:729] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.191596 8411 consensus_queue.cc:260] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.191689 8415 tablet_bootstrap.cc:492] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: No bootstrap required, opened a new log
I20260925 13:12:48.191695 8411 ts_tablet_manager.cc:1428] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.191733 8415 ts_tablet_manager.cc:1397] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:48.191753 8410 tablet_bootstrap.cc:654] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.191715 8420 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.191812 8420 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.191874 8415 raft_consensus.cc:348] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.191926 8415 raft_consensus.cc:374] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.191951 8415 raft_consensus.cc:729] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cfde93644544ad9195cc2a1f847b62, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.191984 8415 consensus_queue.cc:260] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.192073 8415 ts_tablet_manager.cc:1428] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.192107 8418 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.192164 8418 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.192528 8410 tablet_bootstrap.cc:492] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: No bootstrap required, opened a new log
I20260925 13:12:48.192603 8410 ts_tablet_manager.cc:1397] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:48.192765 8410 raft_consensus.cc:348] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.192836 8410 raft_consensus.cc:374] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.192868 8410 raft_consensus.cc:729] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.192917 8410 consensus_queue.cc:260] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.193017 8410 ts_tablet_manager.cc:1428] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:48.193068 8414 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.193177 8414 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.394870 8414 raft_consensus.cc:482] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:48.394987 8414 raft_consensus.cc:504] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.395470 8414 leader_election.cc:288] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:48.398532 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.395736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45593 (local address 127.0.98.1:41101)
0925 13:12:48.395784 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.395786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.395915 (+ 129us) server_negotiation.cc:407] Connection header received
0925 13:12:48.396030 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.396032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.396049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.396132 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:48.396289 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.396787 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.397759 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.398199 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.398263 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.398273 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.398428 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.398466 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:12:48.398548 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.395683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.98.2:45593)
0925 13:12:48.395835 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.395845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.395933 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.396144 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.396145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.396147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:48.396214 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.396215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.396871 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.396872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.397639 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.397641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.398272 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.398273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.398341 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.398360 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.398424 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.398463 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260925 13:12:48.398732 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.395726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41005 (local address 127.0.98.2:40727)
0925 13:12:48.395945 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.395954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.396008 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.396204 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.396205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.396207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:12:48.396277 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.396279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.396975 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.396976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.397812 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.397814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.398452 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.398453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.398529 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.398534 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.398605 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.398646 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":57,"threads_started":1}
I20260925 13:12:48.398715 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.395814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40727 (local address 127.0.98.3:41005)
0925 13:12:48.395876 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.395883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.396009 (+ 126us) server_negotiation.cc:407] Connection header received
0925 13:12:48.396079 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.396080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.396090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.396154 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:12:48.396368 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.396869 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.397914 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.398347 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.398448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.398458 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.398611 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.398650 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:12:48.399042 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489065f7fff04e63aaffa0176effd0fc" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62" is_pre_election: true
I20260925 13:12:48.399097 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489065f7fff04e63aaffa0176effd0fc" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761fca4a37bc434b95d03902d02f4419" is_pre_election: true
I20260925 13:12:48.399152 8336 raft_consensus.cc:2454] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 0.
I20260925 13:12:48.399183 8056 raft_consensus.cc:2454] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 0.
I20260925 13:12:48.399358 8130 leader_election.cc:302] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cfde93644544ad9195cc2a1f847b62, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.399500 8414 raft_consensus.cc:2781] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:48.399540 8414 raft_consensus.cc:482] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:48.399569 8414 raft_consensus.cc:3037] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.400234 8414 raft_consensus.cc:504] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.400383 8414 leader_election.cc:288] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 election: Requested vote from peers 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:48.400554 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489065f7fff04e63aaffa0176effd0fc" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761fca4a37bc434b95d03902d02f4419"
I20260925 13:12:48.400528 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489065f7fff04e63aaffa0176effd0fc" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62"
I20260925 13:12:48.400635 8336 raft_consensus.cc:3037] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.400648 8056 raft_consensus.cc:3037] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.401212 8056 raft_consensus.cc:2454] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 1.
I20260925 13:12:48.401372 8336 raft_consensus.cc:2454] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 1.
I20260925 13:12:48.401460 8129 leader_election.cc:302] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761fca4a37bc434b95d03902d02f4419, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.401580 8414 raft_consensus.cc:2781] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:48.401754 8414 raft_consensus.cc:686] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Running, Role: LEADER
I20260925 13:12:48.401839 8414 consensus_queue.cc:237] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } }
I20260925 13:12:48.402226 8452 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: TxnStatusTablet state changed. Reason: New leader 8584763e51c2480582cb404ce2dc63c5. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.402324 8452 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:48.402483 8454 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 13:12:48.402601 8454 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 13:12:48.402668 7899 catalog_manager.cc:5495] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } health_report { overall_health: HEALTHY } } }
I20260925 13:12:48.419507 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.416936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.0.98.2:35053)
0925 13:12:48.416977 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.416979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.416988 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:48.417074 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.417075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.417093 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.417145 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.417289 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.417818 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.418681 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.418889 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.418962 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.419150 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.419256 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.419338 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.419393 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.419413 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:48.419512 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.416858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.0.1:51658)
0925 13:12:48.416935 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.416946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.416993 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.417165 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.417166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.417167 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:48.417215 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.417217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.417911 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.417913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.418582 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.418584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.418962 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.418963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.419030 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.419050 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.419324 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.419333 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.419388 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.419437 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:12:48.420594 8336 raft_consensus.cc:1261] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Refusing update from remote peer 8584763e51c2480582cb404ce2dc63c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.420733 8056 raft_consensus.cc:1261] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Refusing update from remote peer 8584763e51c2480582cb404ce2dc63c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.420846 8414 consensus_queue.cc:1035] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.420954 8452 consensus_queue.cc:1035] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.421559 8420 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: TxnStatusTablet state changed. Reason: New leader 8584763e51c2480582cb404ce2dc63c5. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.421648 8420 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.421612 8418 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: TxnStatusTablet state changed. Reason: New leader 8584763e51c2480582cb404ce2dc63c5. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.421727 8418 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.422061 8414 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.422147 8414 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:48.422416 8424 mvcc.cc:205] Tried to move back new op lower bound from 7333240702648872960 to 7333240702574534656. Current Snapshot: MvccSnapshot[applied={T|T < 7333240702648872960}]
I20260925 13:12:48.422503 8418 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.422609 8418 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 35cfde93644544ad9195cc2a1f847b62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.423928 8414 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.424012 8414 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 8584763e51c2480582cb404ce2dc63c5: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:12:48.424783 8420 tablet_replica.cc:439] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } }
I20260925 13:12:48.424865 8420 tablet_replica.cc:442] T 489065f7fff04e63aaffa0176effd0fc P 761fca4a37bc434b95d03902d02f4419: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:12:48.426424 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.423675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.0.98.3:41005)
0925 13:12:48.423713 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.423715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.423756 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:12:48.423834 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.423835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.423848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.423927 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.424095 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.424662 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.425601 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.425941 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.426010 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.426113 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.426233 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.426297 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.426321 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.426358 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:12:48.431136 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.428649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37515 (local address 127.0.98.2:35053)
0925 13:12:48.428685 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.428687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.428806 (+ 119us) server_negotiation.cc:407] Connection header received
0925 13:12:48.428864 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.428865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.428873 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.428912 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.429036 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.429450 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.430517 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.430683 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.430730 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.430938 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.430961 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.431009 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.431060 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.431071 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:12:48.431375 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.428588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.98.3:37515)
0925 13:12:48.428745 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.428754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.428809 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.428924 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.428925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.428926 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:48.428973 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.428974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.429521 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.429522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.430447 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.430449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.430774 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.430775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.430854 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.430867 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.431011 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.431013 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.431058 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.431115 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260925 13:12:48.433329 8196 raft_consensus.cc:1261] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Refusing update from remote peer 35cfde93644544ad9195cc2a1f847b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.433332 8056 raft_consensus.cc:1261] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Refusing update from remote peer 35cfde93644544ad9195cc2a1f847b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.433557 8439 consensus_queue.cc:1035] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.433661 8418 consensus_queue.cc:1035] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.442174 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.439265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44143 (local address 127.0.98.3:41005)
0925 13:12:48.439299 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.439301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.439442 (+ 141us) server_negotiation.cc:407] Connection header received
0925 13:12:48.439522 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.439523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.439533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.439583 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.439690 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.440117 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.441373 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.441589 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.441694 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.441869 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.441903 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.441931 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.442073 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.442104 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:12:48.442221 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.439242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41005 (local address 127.0.98.2:44143)
0925 13:12:48.439383 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.439393 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.439465 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.439584 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.439584 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.439586 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:48.439642 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.439644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.440194 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.440195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.441217 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.441220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.441684 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.441685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.441753 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.441767 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.441980 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.441981 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.442025 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.442092 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260925 13:12:48.451910 7897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:48.452037 7897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:48.452732 7897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Sending DeleteTablet for 3 replicas of tablet 7f792512703244afaf3aede0737f5933
I20260925 13:12:48.453008 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f792512703244afaf3aede0737f5933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55028
I20260925 13:12:48.453070 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f792512703244afaf3aede0737f5933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51642
I20260925 13:12:48.453194 8432 tablet_replica.cc:330] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: stopping tablet replica
I20260925 13:12:48.453267 8432 raft_consensus.cc:2229] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.453315 8432 raft_consensus.cc:2258] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.453388 8433 tablet_replica.cc:330] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: stopping tablet replica
I20260925 13:12:48.453447 8433 raft_consensus.cc:2229] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.453490 8433 raft_consensus.cc:2258] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.453981 8433 ts_tablet_manager.cc:1905] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.455683 8433 ts_tablet_manager.cc:1918] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:48.455754 8433 log.cc:1196] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/7f792512703244afaf3aede0737f5933
I20260925 13:12:48.455986 8433 ts_tablet_manager.cc:1939] T 7f792512703244afaf3aede0737f5933 P 8584763e51c2480582cb404ce2dc63c5: Deleting consensus metadata
I20260925 13:12:48.456259 8432 ts_tablet_manager.cc:1905] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.457864 7884 catalog_manager.cc:4841] TS 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053): tablet 7f792512703244afaf3aede0737f5933 (table test-table [id=1af16532c4534b03bd377060e1af4e71]) successfully deleted
I20260925 13:12:48.457973 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f792512703244afaf3aede0737f5933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52204
I20260925 13:12:48.458187 8434 tablet_replica.cc:330] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: stopping tablet replica
I20260925 13:12:48.458271 8434 raft_consensus.cc:2229] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:48.458369 8434 raft_consensus.cc:2258] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.458846 8434 ts_tablet_manager.cc:1905] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.460357 8434 ts_tablet_manager.cc:1918] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:48.460421 8434 log.cc:1196] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/7f792512703244afaf3aede0737f5933
I20260925 13:12:48.460659 8434 ts_tablet_manager.cc:1939] T 7f792512703244afaf3aede0737f5933 P 35cfde93644544ad9195cc2a1f847b62: Deleting consensus metadata
I20260925 13:12:48.463346 8432 ts_tablet_manager.cc:1918] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:48.463403 8432 log.cc:1196] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/7f792512703244afaf3aede0737f5933
I20260925 13:12:48.463613 8432 ts_tablet_manager.cc:1939] T 7f792512703244afaf3aede0737f5933 P 761fca4a37bc434b95d03902d02f4419: Deleting consensus metadata
I20260925 13:12:48.463917 7885 catalog_manager.cc:4841] TS 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101): tablet 7f792512703244afaf3aede0737f5933 (table test-table [id=1af16532c4534b03bd377060e1af4e71]) successfully deleted
I20260925 13:12:48.464166 7886 catalog_manager.cc:4841] TS 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005): tablet 7f792512703244afaf3aede0737f5933 (table test-table [id=1af16532c4534b03bd377060e1af4e71]) successfully deleted
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 13:12:48.491185 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:48.484385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.0.98.62:39359)
0925 13:12:48.484428 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.484430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.484580 (+ 150us) server_negotiation.cc:407] Connection header received
0925 13:12:48.484641 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.484642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.484654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.484720 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:48.485435 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.486249 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.487343 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.487613 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.487692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.491115 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33036: BlockingRecv error: recv got EOF from 127.0.0.1:33036 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:12:48.500790 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.491961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.0.98.62:39359)
0925 13:12:48.491999 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.492001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.492241 (+ 240us) server_negotiation.cc:407] Connection header received
0925 13:12:48.492288 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.492289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.492300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.492361 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:48.492928 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.493667 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.494738 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.495022 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.495095 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.497728 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:48.497729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:48.497729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:48.497737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:48.499455 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:48.499857 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:48.499858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:48.499859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:48.499922 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:48.500147 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:48.500148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:48.500148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:48.500282 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:48.500382 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.500519 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.500602 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:12:48.502461 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:48.502631 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:48.503810 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 99a05dcab18f49348bcdcc9e4d05b993 (DEFAULT_TABLE table=test-table [id=764d05fd882c4e7f9ff874ce2b83270a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:48.503940 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a05dcab18f49348bcdcc9e4d05b993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.503988 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 99a05dcab18f49348bcdcc9e4d05b993 (DEFAULT_TABLE table=test-table [id=764d05fd882c4e7f9ff874ce2b83270a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:48.504124 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a05dcab18f49348bcdcc9e4d05b993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.505362 8411 tablet_bootstrap.cc:492] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Bootstrap starting.
I20260925 13:12:48.505378 8316 tablet_service.cc:1467] Processing CreateTablet for tablet 99a05dcab18f49348bcdcc9e4d05b993 (DEFAULT_TABLE table=test-table [id=764d05fd882c4e7f9ff874ce2b83270a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:48.505486 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a05dcab18f49348bcdcc9e4d05b993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.505671 8411 tablet_bootstrap.cc:654] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.506644 8411 tablet_bootstrap.cc:492] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: No bootstrap required, opened a new log
I20260925 13:12:48.506686 8411 ts_tablet_manager.cc:1397] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:48.506726 8410 tablet_bootstrap.cc:492] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Bootstrap starting.
I20260925 13:12:48.506788 8411 raft_consensus.cc:348] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.506865 8411 raft_consensus.cc:374] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.506887 8411 raft_consensus.cc:729] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.506929 8411 consensus_queue.cc:260] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.507014 8411 ts_tablet_manager.cc:1428] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.507125 8410 tablet_bootstrap.cc:654] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.507429 8415 tablet_bootstrap.cc:492] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Bootstrap starting.
I20260925 13:12:48.507802 8415 tablet_bootstrap.cc:654] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.509312 8410 tablet_bootstrap.cc:492] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: No bootstrap required, opened a new log
I20260925 13:12:48.509358 8410 ts_tablet_manager.cc:1397] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 13:12:48.509493 8410 raft_consensus.cc:348] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.509552 8410 raft_consensus.cc:374] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.509572 8410 raft_consensus.cc:729] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.509609 8410 consensus_queue.cc:260] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.509723 8410 ts_tablet_manager.cc:1428] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.510355 8415 tablet_bootstrap.cc:492] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: No bootstrap required, opened a new log
I20260925 13:12:48.510403 8415 ts_tablet_manager.cc:1397] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 13:12:48.510535 8415 raft_consensus.cc:348] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.510594 8415 raft_consensus.cc:374] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.510613 8415 raft_consensus.cc:729] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cfde93644544ad9195cc2a1f847b62, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.510656 8415 consensus_queue.cc:260] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.510732 8415 ts_tablet_manager.cc:1428] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.735920 8452 raft_consensus.cc:482] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:48.736027 8452 raft_consensus.cc:504] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.736209 8452 leader_election.cc:288] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:48.736428 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a05dcab18f49348bcdcc9e4d05b993" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62" is_pre_election: true
I20260925 13:12:48.736428 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a05dcab18f49348bcdcc9e4d05b993" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761fca4a37bc434b95d03902d02f4419" is_pre_election: true
I20260925 13:12:48.736522 8336 raft_consensus.cc:2454] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 0.
I20260925 13:12:48.736522 8056 raft_consensus.cc:2454] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 0.
I20260925 13:12:48.736655 8129 leader_election.cc:302] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761fca4a37bc434b95d03902d02f4419, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.736768 8452 raft_consensus.cc:2781] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:48.736835 8452 raft_consensus.cc:482] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:48.736851 8452 raft_consensus.cc:3037] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.737562 8452 raft_consensus.cc:504] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.737756 8452 leader_election.cc:288] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 election: Requested vote from peers 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:48.737893 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a05dcab18f49348bcdcc9e4d05b993" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761fca4a37bc434b95d03902d02f4419"
I20260925 13:12:48.737891 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a05dcab18f49348bcdcc9e4d05b993" candidate_uuid: "8584763e51c2480582cb404ce2dc63c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62"
I20260925 13:12:48.737999 8336 raft_consensus.cc:3037] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.737967 8056 raft_consensus.cc:3037] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.738586 8056 raft_consensus.cc:2454] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 1.
I20260925 13:12:48.738768 8129 leader_election.cc:302] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761fca4a37bc434b95d03902d02f4419, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.738830 8336 raft_consensus.cc:2454] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8584763e51c2480582cb404ce2dc63c5 in term 1.
I20260925 13:12:48.738943 8452 raft_consensus.cc:2781] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:48.738997 8452 raft_consensus.cc:686] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Running, Role: LEADER
I20260925 13:12:48.739042 8452 consensus_queue.cc:237] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.739666 7899 catalog_manager.cc:5495] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8584763e51c2480582cb404ce2dc63c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:48.791093 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.788841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.98.2:35053)
0925 13:12:48.788910 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.788911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.788920 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:48.788998 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.788999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.789008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.789055 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.789177 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.789627 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.790504 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.790694 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.790746 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.790859 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.790901 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.790952 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.790994 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.791008 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 13:12:48.792013 8336 raft_consensus.cc:1261] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Refusing update from remote peer 8584763e51c2480582cb404ce2dc63c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.792038 8056 raft_consensus.cc:1261] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Refusing update from remote peer 8584763e51c2480582cb404ce2dc63c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.792215 8468 consensus_queue.cc:1035] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.792310 8452 consensus_queue.cc:1035] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.795383 7897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:48.795495 7897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:48.796082 7897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Sending DeleteTablet for 3 replicas of tablet 99a05dcab18f49348bcdcc9e4d05b993
I20260925 13:12:48.796298 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a05dcab18f49348bcdcc9e4d05b993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55028
I20260925 13:12:48.796403 8432 tablet_replica.cc:330] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: stopping tablet replica
I20260925 13:12:48.796464 8432 raft_consensus.cc:2229] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.796502 8432 raft_consensus.cc:2258] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.796615 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a05dcab18f49348bcdcc9e4d05b993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51642
I20260925 13:12:48.796697 8433 tablet_replica.cc:330] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: stopping tablet replica
I20260925 13:12:48.796751 8433 raft_consensus.cc:2229] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:48.796835 8433 raft_consensus.cc:2258] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.796990 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:12:48.797147 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:12:48.797405 8433 ts_tablet_manager.cc:1905] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.798096 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a05dcab18f49348bcdcc9e4d05b993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52204
I20260925 13:12:48.798188 8434 tablet_replica.cc:330] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: stopping tablet replica
I20260925 13:12:48.798228 8434 raft_consensus.cc:2229] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.798269 8434 raft_consensus.cc:2258] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.798331 8432 ts_tablet_manager.cc:1905] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.798902 8433 ts_tablet_manager.cc:1918] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:48.798970 8433 log.cc:1196] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/99a05dcab18f49348bcdcc9e4d05b993
I20260925 13:12:48.799185 8433 ts_tablet_manager.cc:1939] T 99a05dcab18f49348bcdcc9e4d05b993 P 8584763e51c2480582cb404ce2dc63c5: Deleting consensus metadata
I20260925 13:12:48.799547 7884 catalog_manager.cc:4841] TS 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053): tablet 99a05dcab18f49348bcdcc9e4d05b993 (table test-table [id=764d05fd882c4e7f9ff874ce2b83270a]) successfully deleted
I20260925 13:12:48.799793 8432 ts_tablet_manager.cc:1918] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:48.799846 8432 log.cc:1196] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/99a05dcab18f49348bcdcc9e4d05b993
I20260925 13:12:48.800040 8432 ts_tablet_manager.cc:1939] T 99a05dcab18f49348bcdcc9e4d05b993 P 761fca4a37bc434b95d03902d02f4419: Deleting consensus metadata
I20260925 13:12:48.800541 8036 tablet_service.cc:1467] Processing CreateTablet for tablet f25d9a21224f401289badf5892423969 (DEFAULT_TABLE table=test-table [id=5c999244d13d44f6b97b57ea74670153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:48.800554 8176 tablet_service.cc:1467] Processing CreateTablet for tablet f25d9a21224f401289badf5892423969 (DEFAULT_TABLE table=test-table [id=5c999244d13d44f6b97b57ea74670153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:48.800639 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25d9a21224f401289badf5892423969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.800648 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25d9a21224f401289badf5892423969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.801714 8410 tablet_bootstrap.cc:492] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Bootstrap starting.
I20260925 13:12:48.801719 8411 tablet_bootstrap.cc:492] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419: Bootstrap starting.
I20260925 13:12:48.801863 8316 tablet_service.cc:1467] Processing CreateTablet for tablet f25d9a21224f401289badf5892423969 (DEFAULT_TABLE table=test-table [id=5c999244d13d44f6b97b57ea74670153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:12:48.801951 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25d9a21224f401289badf5892423969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:48.801993 8411 tablet_bootstrap.cc:654] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.802006 8410 tablet_bootstrap.cc:654] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.802632 8410 tablet_bootstrap.cc:492] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: No bootstrap required, opened a new log
I20260925 13:12:48.802699 8410 ts_tablet_manager.cc:1397] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:48.802796 8410 raft_consensus.cc:348] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.802860 8410 raft_consensus.cc:374] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.802879 8410 raft_consensus.cc:729] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8584763e51c2480582cb404ce2dc63c5, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.802918 8410 consensus_queue.cc:260] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.803020 8410 ts_tablet_manager.cc:1428] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:48.803296 8415 tablet_bootstrap.cc:492] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Bootstrap starting.
I20260925 13:12:48.803314 7885 catalog_manager.cc:4841] TS 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101): tablet 99a05dcab18f49348bcdcc9e4d05b993 (table test-table [id=764d05fd882c4e7f9ff874ce2b83270a]) successfully deleted
I20260925 13:12:48.803596 8415 tablet_bootstrap.cc:654] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:48.803835 8434 ts_tablet_manager.cc:1905] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.804171 8415 tablet_bootstrap.cc:492] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: No bootstrap required, opened a new log
I20260925 13:12:48.804221 8415 ts_tablet_manager.cc:1397] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:12:48.804307 8415 raft_consensus.cc:348] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.804371 8415 raft_consensus.cc:374] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.804392 8415 raft_consensus.cc:729] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cfde93644544ad9195cc2a1f847b62, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.804440 8415 consensus_queue.cc:260] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.804513 8415 ts_tablet_manager.cc:1428] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.805145 8434 ts_tablet_manager.cc:1918] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:12:48.805212 8434 log.cc:1196] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/99a05dcab18f49348bcdcc9e4d05b993
I20260925 13:12:48.805411 8434 ts_tablet_manager.cc:1939] T 99a05dcab18f49348bcdcc9e4d05b993 P 35cfde93644544ad9195cc2a1f847b62: Deleting consensus metadata
I20260925 13:12:48.805716 7886 catalog_manager.cc:4841] TS 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005): tablet 99a05dcab18f49348bcdcc9e4d05b993 (table test-table [id=764d05fd882c4e7f9ff874ce2b83270a]) successfully deleted
I20260925 13:12:48.805964 8411 tablet_bootstrap.cc:492] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419: No bootstrap required, opened a new log
I20260925 13:12:48.806025 8411 ts_tablet_manager.cc:1397] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260925 13:12:48.806136 8411 raft_consensus.cc:348] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.806201 8411 raft_consensus.cc:374] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:48.806231 8411 raft_consensus.cc:729] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Initialized, Role: FOLLOWER
I20260925 13:12:48.806281 8411 consensus_queue.cc:260] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.806360 8411 ts_tablet_manager.cc:1428] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:48.834214 8420 raft_consensus.cc:482] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:12:48.834336 8420 raft_consensus.cc:504] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.834540 8420 leader_election.cc:288] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:48.834714 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25d9a21224f401289badf5892423969" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8584763e51c2480582cb404ce2dc63c5" is_pre_election: true
I20260925 13:12:48.834738 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25d9a21224f401289badf5892423969" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62" is_pre_election: true
I20260925 13:12:48.834827 8196 raft_consensus.cc:2454] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 0.
I20260925 13:12:48.834838 8336 raft_consensus.cc:2454] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 0.
I20260925 13:12:48.834968 7988 leader_election.cc:302] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761fca4a37bc434b95d03902d02f4419, 8584763e51c2480582cb404ce2dc63c5; no voters:
I20260925 13:12:48.835109 8420 raft_consensus.cc:2781] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:12:48.835160 8420 raft_consensus.cc:482] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:12:48.835179 8420 raft_consensus.cc:3037] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.836052 8420 raft_consensus.cc:504] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.836181 8420 leader_election.cc:288] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 election: Requested vote from peers 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053), 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005)
I20260925 13:12:48.836305 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25d9a21224f401289badf5892423969" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8584763e51c2480582cb404ce2dc63c5"
I20260925 13:12:48.836383 8196 raft_consensus.cc:3037] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.836364 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25d9a21224f401289badf5892423969" candidate_uuid: "761fca4a37bc434b95d03902d02f4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cfde93644544ad9195cc2a1f847b62"
I20260925 13:12:48.836436 8336 raft_consensus.cc:3037] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:48.837121 8196 raft_consensus.cc:2454] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 1.
I20260925 13:12:48.837121 8336 raft_consensus.cc:2454] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761fca4a37bc434b95d03902d02f4419 in term 1.
I20260925 13:12:48.837275 7990 leader_election.cc:302] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cfde93644544ad9195cc2a1f847b62, 761fca4a37bc434b95d03902d02f4419; no voters:
I20260925 13:12:48.837355 8420 raft_consensus.cc:2781] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:48.837459 8420 raft_consensus.cc:686] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 1 LEADER]: Becoming Leader. State: Replica: 761fca4a37bc434b95d03902d02f4419, State: Running, Role: LEADER
I20260925 13:12:48.837530 8420 consensus_queue.cc:237] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } }
I20260925 13:12:48.838169 7897 catalog_manager.cc:5495] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 reported cstate change: term changed from 0 to 1, leader changed from <none> to 761fca4a37bc434b95d03902d02f4419 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "761fca4a37bc434b95d03902d02f4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761fca4a37bc434b95d03902d02f4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20260925 13:12:48.845184 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.842868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.1:41101)
0925 13:12:48.842904 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.842906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.842915 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:48.842996 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.842997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.843005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.843059 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.843153 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.843881 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.844537 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.844833 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.844898 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.844987 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.845018 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.845061 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.845103 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.845119 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:12:48.849858 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.847514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56833 (local address 127.0.98.2:35053)
0925 13:12:48.847556 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.847558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.847698 (+ 140us) server_negotiation.cc:407] Connection header received
0925 13:12:48.847759 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.847760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.847771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.847824 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.847925 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.848379 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.849125 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.849326 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.849403 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.849575 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.849606 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.849720 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.849780 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.849795 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:12:48.849890 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.847474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.98.1:56833)
0925 13:12:48.847623 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.847632 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.847711 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.847827 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.847828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.847829 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:48.847880 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.847881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.848461 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.848462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.849033 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.849035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.849403 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.849403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.849458 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.849472 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.849724 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.849726 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.849776 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.849818 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260925 13:12:48.851858 8196 raft_consensus.cc:1261] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Refusing update from remote peer 761fca4a37bc434b95d03902d02f4419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.852007 8336 raft_consensus.cc:1261] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Refusing update from remote peer 761fca4a37bc434b95d03902d02f4419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:12:48.852059 8420 consensus_queue.cc:1035] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8584763e51c2480582cb404ce2dc63c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.852205 8420 consensus_queue.cc:1035] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cfde93644544ad9195cc2a1f847b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:12:48.859184 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.856455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.98.2:57197)
0925 13:12:48.856496 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.856505 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:48.856543 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:48.856941 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:48.856942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:48.856943 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:12:48.856981 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.856982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.857597 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.857598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.858402 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:48.858404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.858806 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.858806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:48.858853 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.858859 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.859029 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.859031 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:12:48.859078 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:12:48.859130 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:12:48.859179 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.856792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57197 (local address 127.0.98.1:41101)
0925 13:12:48.856822 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:48.856824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:48.856831 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:48.856850 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:48.856851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:48.856859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:48.856905 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:12:48.857070 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.857518 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.858493 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:48.858722 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:48.858806 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:48.858933 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:12:48.858968 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:12:48.859029 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:48.859102 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 13:12:48.859120 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:12:48.865206 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:48.865317 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:12:48.865965 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64849755469b4710b2ff4acc02f25a54: Sending DeleteTablet for 3 replicas of tablet f25d9a21224f401289badf5892423969
I20260925 13:12:48.866207 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet f25d9a21224f401289badf5892423969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51642
I20260925 13:12:48.866222 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet f25d9a21224f401289badf5892423969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55028
I20260925 13:12:48.866230 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet f25d9a21224f401289badf5892423969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:12:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52204
I20260925 13:12:48.866317 8432 tablet_replica.cc:330] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419: stopping tablet replica
I20260925 13:12:48.866321 8433 tablet_replica.cc:330] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: stopping tablet replica
I20260925 13:12:48.866371 8432 raft_consensus.cc:2229] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:12:48.866374 8433 raft_consensus.cc:2229] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.866405 8433 raft_consensus.cc:2258] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.866423 8432 raft_consensus.cc:2258] T f25d9a21224f401289badf5892423969 P 761fca4a37bc434b95d03902d02f4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.866508 8434 tablet_replica.cc:330] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: stopping tablet replica
I20260925 13:12:48.866559 8434 raft_consensus.cc:2229] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:12:48.866590 8434 raft_consensus.cc:2258] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:12:48.866811 8433 ts_tablet_manager.cc:1905] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.866937 8434 ts_tablet_manager.cc:1905] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:12:48.867053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 7971
I20260925 13:12:48.869362 8433 ts_tablet_manager.cc:1918] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:48.869424 8433 log.cc:1196] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/f25d9a21224f401289badf5892423969
I20260925 13:12:48.869684 8433 ts_tablet_manager.cc:1939] T f25d9a21224f401289badf5892423969 P 8584763e51c2480582cb404ce2dc63c5: Deleting consensus metadata
I20260925 13:12:48.869956 8434 ts_tablet_manager.cc:1918] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:12:48.870003 8434 log.cc:1196] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/f25d9a21224f401289badf5892423969
I20260925 13:12:48.869999 7884 catalog_manager.cc:4841] TS 8584763e51c2480582cb404ce2dc63c5 (127.0.98.2:35053): tablet f25d9a21224f401289badf5892423969 (table test-table [id=5c999244d13d44f6b97b57ea74670153]) successfully deleted
I20260925 13:12:48.870160 8434 ts_tablet_manager.cc:1939] T f25d9a21224f401289badf5892423969 P 35cfde93644544ad9195cc2a1f847b62: Deleting consensus metadata
I20260925 13:12:48.870426 7886 catalog_manager.cc:4841] TS 35cfde93644544ad9195cc2a1f847b62 (127.0.98.3:41005): tablet f25d9a21224f401289badf5892423969 (table test-table [id=5c999244d13d44f6b97b57ea74670153]) successfully deleted
W20260925 13:12:48.874549 7885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41101 (error 108)
I20260925 13:12:48.874853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 8111
I20260925 13:12:48.875049 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:48.874897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.0.1:55058)
0925 13:12:48.874979 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:48.875012 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":34,"threads_started":1}
W20260925 13:12:48.875128 7885 catalog_manager.cc:4584] TS 761fca4a37bc434b95d03902d02f4419 (127.0.98.1:41101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f25d9a21224f401289badf5892423969: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41101: connect: Connection refused (error 111)
I20260925 13:12:48.881953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 8251
I20260925 13:12:48.887181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 7867
2026-09-25T13:12:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2829 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[8506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[8506](info): set up 2 sockets
Sep 25 13:12:48 dist-test-slave-2d0s krb5kdc[8506](info): commencing operation
krb5kdc: starting...
W20260925 13:12:49.906453 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:49 dist-test-slave-2d0s krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:50.024765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:50.103012 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:50.103211 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:50.103232 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:50.104262 8519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:50.104293 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:50.104307 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:50.104319 8519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:50.104331 8519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:50.105700 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:50.105875 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:50.106021 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:50.110591 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:50.110625 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:50.110598 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:50.110992 8519 server_base.cc:1034] running on GCE node
I20260925 13:12:50.111209 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:50.111459 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:50.112603 8519 hybrid_clock.cc:648] HybridClock initialized: now 1790341970112584 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T13:12:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1234 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:12:50 dist-test-slave-2d0s krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:12:50 dist-test-slave-2d0s krb5kdc[8532](info): set up 2 sockets
Sep 25 13:12:50 dist-test-slave-2d0s krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260925 13:12:51.138223 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:12:51.247655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35615
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:51.327348 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:51.327553 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:51.327574 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:51.328745 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:51.328787 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:51.328801 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:51.328814 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:51.328873 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:51.330368 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:51.330612 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:51.330858 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:51.335594 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.335587 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.335599 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:51.335917 8545 server_base.cc:1034] running on GCE node
I20260925 13:12:51.336149 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:51.336442 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:51.337633 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790341971337622 us; error 31 us; skew 500 ppm
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:51.345371 8545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:51.345873 8545 webserver.cc:460] Webserver started at http://127.0.98.62:42693/ using document root <none> and password file <none>
I20260925 13:12:51.346065 8545 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:51.346107 8545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:51.346200 8545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:51.347234 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "eedee2c2bf1d4a689562f5d504f309b0"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "62a244e2b4a513b43073d02e59bd55f1"
server_key_iv: "f2dcaae8c0ab89b1bd28314b1afb63e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.347532 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "eedee2c2bf1d4a689562f5d504f309b0"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "62a244e2b4a513b43073d02e59bd55f1"
server_key_iv: "f2dcaae8c0ab89b1bd28314b1afb63e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.348805 8545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:12:51.349474 8560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.349645 8545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:51.349786 8545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "eedee2c2bf1d4a689562f5d504f309b0"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "62a244e2b4a513b43073d02e59bd55f1"
server_key_iv: "f2dcaae8c0ab89b1bd28314b1afb63e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.349871 8545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:51.371074 8545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:51.372983 8545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:51.373152 8545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:51.377184 8545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35615
I20260925 13:12:51.377221 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35615 every 8 connection(s)
I20260925 13:12:51.377768 8545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:51.378494 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:12:51.380720 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Bootstrap starting.
I20260925 13:12:51.381254 8613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Neither blocks nor log segments found. Creating new log.
I20260925 13:12:51.381507 8613 log.cc:824] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:51.382194 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: No bootstrap required, opened a new log
I20260925 13:12:51.382481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 8545
I20260925 13:12:51.382593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:12:51.382787 392 external_mini_cluster.cc:1428] Setting key 48886ec89e8f399e1a59fa0473977fdb
I20260925 13:12:51.384792 8613 raft_consensus.cc:348] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:51.384968 8613 raft_consensus.cc:374] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:12:51.385008 8613 raft_consensus.cc:729] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedee2c2bf1d4a689562f5d504f309b0, State: Initialized, Role: FOLLOWER
I20260925 13:12:51.385191 8613 consensus_queue.cc:260] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:51.385283 8613 raft_consensus.cc:388] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:51.385332 8613 raft_consensus.cc:482] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:51.385387 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:12:51.386089 8613 raft_consensus.cc:504] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:51.386176 8613 leader_election.cc:302] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eedee2c2bf1d4a689562f5d504f309b0; no voters:
I20260925 13:12:51.386380 8613 leader_election.cc:288] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:12:51.386404 8618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:12:51.386508 8618 raft_consensus.cc:686] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 LEADER]: Becoming Leader. State: Replica: eedee2c2bf1d4a689562f5d504f309b0, State: Running, Role: LEADER
I20260925 13:12:51.386664 8613 sys_catalog.cc:564] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:51.386648 8618 consensus_queue.cc:237] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:51.387250 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eedee2c2bf1d4a689562f5d504f309b0. Latest consensus state: current_term: 1 leader_uuid: "eedee2c2bf1d4a689562f5d504f309b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } } }
I20260925 13:12:51.387260 8619 sys_catalog.cc:455] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eedee2c2bf1d4a689562f5d504f309b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } } }
I20260925 13:12:51.387447 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:51.387516 8619 sys_catalog.cc:458] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:51.387763 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:51.388293 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:51.390031 8627 catalog_manager.cc:1347] Generated new cluster ID: a98b004bf38944e783bab3644d8f26f2
I20260925 13:12:51.390094 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:51.395799 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.383298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.0.98.62:35615)
0925 13:12:51.383470 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:51.383478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:51.383503 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:12:51.384774 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:51.384780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:51.384827 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:51.384940 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:51.385769 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.386946 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.388060 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.388715 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.388852 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.391751 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:51.391755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:51.391756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:51.391768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:51.393649 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.394058 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.394060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.394061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.394176 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.394373 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.394373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.394374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.394713 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:51.394869 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:51.395414 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 13:12:51.395579 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260925 13:12:51.397168 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:12:51.397645 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:51.407931 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Generated new TSK 0
I20260925 13:12:51.408102 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:12:51.430584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35615
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:51.510102 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:51.510295 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:51.510326 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:51.511408 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:51.511459 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:51.511484 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:51.512888 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:51.513067 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:51.513234 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:51.518055 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.518054 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:51.518188 8639 server_base.cc:1034] running on GCE node
W20260925 13:12:51.518147 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:51.518522 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:51.518805 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:51.519958 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790341971519945 us; error 29 us; skew 500 ppm
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:51.522317 8639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:12:51.522681 8639 webserver.cc:460] Webserver started at http://127.0.98.1:44263/ using document root <none> and password file <none>
I20260925 13:12:51.522861 8639 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:51.522899 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:51.522979 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:51.523936 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "b7abdd5a84174ddb8f6b6eb6381aa792"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "24625af1c79bc54a7f2f49d2f607b7d4"
server_key_iv: "284243779bb7ebe1d083434b6789a26f"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.524207 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b7abdd5a84174ddb8f6b6eb6381aa792"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "24625af1c79bc54a7f2f49d2f607b7d4"
server_key_iv: "284243779bb7ebe1d083434b6789a26f"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.525364 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:51.526029 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.526190 8639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.526278 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "b7abdd5a84174ddb8f6b6eb6381aa792"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "24625af1c79bc54a7f2f49d2f607b7d4"
server_key_iv: "284243779bb7ebe1d083434b6789a26f"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.526345 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:51.547909 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:51.549640 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:51.549856 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:51.550078 8639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:51.550381 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:51.550414 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.550460 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:51.550485 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.556270 8639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39245
I20260925 13:12:51.556365 8767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39245 every 8 connection(s)
I20260925 13:12:51.556646 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:51.565095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 8639
I20260925 13:12:51.565207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:12:51.565400 392 external_mini_cluster.cc:1428] Setting key 0e4870dbedb1ef60550563f8dc2d9dfe
I20260925 13:12:51.565573 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.557360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37375 (local address 127.0.98.62:35615)
0925 13:12:51.557422 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:51.557424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:51.557632 (+ 208us) server_negotiation.cc:407] Connection header received
0925 13:12:51.558604 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:51.558606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:51.558618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:51.558669 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:51.559295 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.559798 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.560790 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.561014 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.561089 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.563033 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:51.563034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:51.563035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:51.563044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:51.564195 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.564582 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.564584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.564584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.564644 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.564881 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.564882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.564882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.565007 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:51.565076 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:51.565398 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 13:12:51.565486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:12:51.565758 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.557344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:37375)
0925 13:12:51.557509 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:51.557548 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:51.558493 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:51.558763 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:51.558768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:51.559017 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:51.559196 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:51.559200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.559902 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.559903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:51.560696 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:51.560698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.561093 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.561095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:51.561227 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.561660 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:51.561667 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:51.562935 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:51.564285 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:51.564291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:51.564293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:51.564509 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:51.564710 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:51.564711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:51.564712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:51.564815 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:51.565078 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:51.565085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:51.565235 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 13:12:51.565373 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 13:12:51.565518 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260925 13:12:51.566246 8768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35615
I20260925 13:12:51.566381 8768 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:51.566546 8768 heartbeater.cc:511] Master 127.0.98.62:35615 requested a full tablet report, sending...
I20260925 13:12:51.567041 8576 ts_manager.cc:194] Registered new tserver with Master: b7abdd5a84174ddb8f6b6eb6381aa792 (127.0.98.1:39245)
I20260925 13:12:51.567971 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37375
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:12:51.590533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35615
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:51.668722 8773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:51.668908 8773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:51.668932 8773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:51.670029 8773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:51.670063 8773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:51.670087 8773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:51.671499 8773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:51.671669 8773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:51.671833 8773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:51.676616 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.676610 8781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.676775 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:51.677060 8773 server_base.cc:1034] running on GCE node
I20260925 13:12:51.677286 8773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:51.677558 8773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:51.678714 8773 hybrid_clock.cc:648] HybridClock initialized: now 1790341971678709 us; error 34 us; skew 500 ppm
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:51.681093 8773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:12:51.681495 8773 webserver.cc:460] Webserver started at http://127.0.98.2:35801/ using document root <none> and password file <none>
I20260925 13:12:51.681777 8773 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:51.681844 8773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:51.681993 8773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:51.682989 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "01f8c7e4079e456db35adf3373ac78b1"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "2be26e2595528d10e300d2f188ae5da4"
server_key_iv: "c8850c0f83fe4a1fd99b3f6ba7d5c970"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.683337 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01f8c7e4079e456db35adf3373ac78b1"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "2be26e2595528d10e300d2f188ae5da4"
server_key_iv: "c8850c0f83fe4a1fd99b3f6ba7d5c970"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.684607 8773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:12:51.685335 8788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.685653 8773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:12:51.685832 8773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "01f8c7e4079e456db35adf3373ac78b1"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "2be26e2595528d10e300d2f188ae5da4"
server_key_iv: "c8850c0f83fe4a1fd99b3f6ba7d5c970"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.685918 8773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:51.700209 8773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:51.702018 8773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:51.702181 8773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:51.702389 8773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:51.702673 8773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:51.702732 8773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.702775 8773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:51.702801 8773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.708312 8773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42771
I20260925 13:12:51.708359 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42771 every 8 connection(s)
I20260925 13:12:51.708650 8773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:51.714882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 8773
I20260925 13:12:51.715005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:12:51.715194 392 external_mini_cluster.cc:1428] Setting key 01c8440fbf78a73ac92af8dba284778e
I20260925 13:12:51.718294 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.709326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53531 (local address 127.0.98.62:35615)
0925 13:12:51.709367 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:51.709369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:51.709620 (+ 251us) server_negotiation.cc:407] Connection header received
0925 13:12:51.710599 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:51.710601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:51.710614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:51.710687 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:51.711272 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.711803 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.712805 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.712992 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.713070 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.715301 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:51.715302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:51.715303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:51.715311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:51.716891 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.717254 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.717255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.717255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.717316 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.717638 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.717643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.717644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.717801 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:51.717855 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:51.718165 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 13:12:51.718234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:12:51.718539 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.709314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:53531)
0925 13:12:51.709489 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:51.709547 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:51.710478 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:51.710762 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:51.710766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:51.711008 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:51.711178 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:51.711182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.711897 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.711898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:51.712708 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:51.712710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.713078 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.713079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:51.713199 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.713803 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:51.713811 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:51.715170 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:51.716977 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:51.716982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:51.716984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:51.717183 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:51.717419 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:51.717422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:51.717422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:51.717562 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:51.717868 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:51.717873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:51.718032 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 13:12:51.718147 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 13:12:51.718308 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260925 13:12:51.718991 8902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35615
I20260925 13:12:51.719094 8902 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:51.719264 8902 heartbeater.cc:511] Master 127.0.98.62:35615 requested a full tablet report, sending...
I20260925 13:12:51.719664 8576 ts_manager.cc:194] Registered new tserver with Master: 01f8c7e4079e456db35adf3373ac78b1 (127.0.98.2:42771)
I20260925 13:12:51.720052 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53531
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:12:51.739952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35615
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:51.817471 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:51.817713 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:51.817747 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:51.818868 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:51.818938 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:51.818980 8907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:51.820437 8907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:51.820688 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:51.820909 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:51.825583 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.825598 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:51.825588 8912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:51.825814 8907 server_base.cc:1034] running on GCE node
I20260925 13:12:51.826050 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:51.826284 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:51.827447 8907 hybrid_clock.cc:648] HybridClock initialized: now 1790341971827436 us; error 27 us; skew 500 ppm
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:51.830098 8907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:12:51.830495 8907 webserver.cc:460] Webserver started at http://127.0.98.3:41479/ using document root <none> and password file <none>
I20260925 13:12:51.830724 8907 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:51.830792 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:51.830943 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:12:51.831984 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4fbbfd7516a4d6abbe59782349137dc"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "a07379f56701420893fb9e2be783d4f9"
server_key_iv: "46ef20d5619cd38fd355df7d37516296"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.832336 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4fbbfd7516a4d6abbe59782349137dc"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "a07379f56701420893fb9e2be783d4f9"
server_key_iv: "46ef20d5619cd38fd355df7d37516296"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.833541 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:12:51.834314 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.834484 8907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:12:51.834578 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "e4fbbfd7516a4d6abbe59782349137dc"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "a07379f56701420893fb9e2be783d4f9"
server_key_iv: "46ef20d5619cd38fd355df7d37516296"
server_key_version: "encryptionkey@0"
I20260925 13:12:51.834669 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:51.845291 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:51.847492 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:51.847636 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:51.847852 8907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:51.848145 8907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:51.848196 8907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.848249 8907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:51.848277 8907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:51.853861 8907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43263
I20260925 13:12:51.853930 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43263 every 8 connection(s)
I20260925 13:12:51.854218 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:51 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:12:51.863325 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.854887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38095 (local address 127.0.98.62:35615)
0925 13:12:51.854926 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:51.854928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:51.855175 (+ 247us) server_negotiation.cc:407] Connection header received
0925 13:12:51.856117 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:51.856118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:51.856131 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:51.856194 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:51.856789 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.857310 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.858345 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.858540 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.858625 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.860553 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:51.860555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:51.860555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:51.860563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:51.861936 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.862369 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.862370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.862371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.862428 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:51.862684 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:51.862685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:51.862685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:51.862802 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:51.862872 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:51.863149 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 13:12:51.863236 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:12:51.863500 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:51.854890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:38095)
0925 13:12:51.855048 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:51.855080 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:51.855997 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:51.856261 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:51.856265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:51.856504 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:51.856692 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:51.856696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.857405 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.857406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:51.858257 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:51.858259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:51.858632 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:51.858633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:51.858761 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:51.859166 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:51.859173 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:51.860423 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:12:51.862038 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:51.862044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:51.862046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:51.862273 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:51.862501 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:12:51.862502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:12:51.862502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:12:51.862605 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:12:51.862875 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:12:51.862882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:12:51.863020 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 13:12:51.863126 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:12:51.863272 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260925 13:12:51.863960 9036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35615
I20260925 13:12:51.864032 9036 heartbeater.cc:464] Registering TS with master...
I20260925 13:12:51.864205 9036 heartbeater.cc:511] Master 127.0.98.62:35615 requested a full tablet report, sending...
I20260925 13:12:51.864323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 8907
I20260925 13:12:51.864437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:12:51.864583 8576 ts_manager.cc:194] Registered new tserver with Master: e4fbbfd7516a4d6abbe59782349137dc (127.0.98.3:43263)
I20260925 13:12:51.864640 392 external_mini_cluster.cc:1428] Setting key 8a5953df4d2b6822b9d1b401cda9fed3
I20260925 13:12:51.865008 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38095
I20260925 13:12:51.865540 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 13:12:51.928679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 8639
I20260925 13:12:51.933463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 8773
I20260925 13:12:51.937628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 8907
I20260925 13:12:51.941833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 8545
I20260925 13:12:51.947119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35615
--webserver_interface=127.0.98.62
--webserver_port=42693
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:12:52.028811 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:52.029007 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:52.029029 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:52.030251 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:12:52.030292 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:52.030305 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:52.030347 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:12:52.030373 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:12:52.031900 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:52.032150 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:52.032383 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:52.036972 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:52.037015 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:52.037220 9045 server_base.cc:1034] running on GCE node
W20260925 13:12:52.036994 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:52.037493 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:52.037827 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:52.038977 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790341972038965 us; error 28 us; skew 500 ppm
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:52.041337 9045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:12:52.041880 9045 webserver.cc:460] Webserver started at http://127.0.98.62:42693/ using document root <none> and password file <none>
I20260925 13:12:52.042124 9045 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:52.042203 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:52.043505 9045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:52.044188 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.044360 9045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:52.044464 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "eedee2c2bf1d4a689562f5d504f309b0"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "62a244e2b4a513b43073d02e59bd55f1"
server_key_iv: "f2dcaae8c0ab89b1bd28314b1afb63e8"
server_key_version: "encryptionkey@0"
I20260925 13:12:52.044914 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:52.066753 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:52.068764 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:52.068931 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:52.073061 9045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35615
I20260925 13:12:52.073099 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35615 every 8 connection(s)
I20260925 13:12:52.073647 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:12:52.074864 9113 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:12:52.075482 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Bootstrap starting.
I20260925 13:12:52.082029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9045
I20260925 13:12:52.082404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39245
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35615
--webserver_port=44263
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:12:52.083689 9113 log.cc:824] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:12:52.085354 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:12:52.085635 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Bootstrap complete.
I20260925 13:12:52.087862 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:52.088109 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedee2c2bf1d4a689562f5d504f309b0, State: Initialized, Role: FOLLOWER
I20260925 13:12:52.088315 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:52.088413 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:12:52.088462 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:12:52.088511 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:12:52.089473 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:52.089593 9113 leader_election.cc:302] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eedee2c2bf1d4a689562f5d504f309b0; no voters:
I20260925 13:12:52.089802 9113 leader_election.cc:288] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:12:52.089865 9117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:12:52.090013 9117 raft_consensus.cc:686] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [term 2 LEADER]: Becoming Leader. State: Replica: eedee2c2bf1d4a689562f5d504f309b0, State: Running, Role: LEADER
I20260925 13:12:52.090103 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:12:52.090121 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } }
I20260925 13:12:52.090423 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eedee2c2bf1d4a689562f5d504f309b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } } }
I20260925 13:12:52.090425 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eedee2c2bf1d4a689562f5d504f309b0. Latest consensus state: current_term: 2 leader_uuid: "eedee2c2bf1d4a689562f5d504f309b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eedee2c2bf1d4a689562f5d504f309b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35615 } } }
I20260925 13:12:52.090620 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:52.090605 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:12:52.090909 9126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:12:52.091403 9126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:12:52.091550 9126 catalog_manager.cc:1259] Loaded cluster ID: a98b004bf38944e783bab3644d8f26f2
I20260925 13:12:52.091826 9126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:12:52.093173 9126 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:12:52.093350 9126 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eedee2c2bf1d4a689562f5d504f309b0: Loaded TSK: 0
I20260925 13:12:52.093555 9126 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:12:52.164263 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:52.164458 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:52.164480 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:52.165647 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:52.165747 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:52.165776 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:12:52.167265 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44263
--tserver_master_addrs=127.0.98.62:35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:52.167496 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:52.167745 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:52.172613 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:52.172793 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:52.172613 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:52.173391 9115 server_base.cc:1034] running on GCE node
I20260925 13:12:52.173602 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:52.173889 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:52.175056 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790341972175036 us; error 36 us; skew 500 ppm
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:52.177757 9115 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:12:52.178263 9115 webserver.cc:460] Webserver started at http://127.0.98.1:44263/ using document root <none> and password file <none>
I20260925 13:12:52.178504 9115 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:52.178596 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:52.179939 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:52.180538 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.180707 9115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:52.180819 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "b7abdd5a84174ddb8f6b6eb6381aa792"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "24625af1c79bc54a7f2f49d2f607b7d4"
server_key_iv: "284243779bb7ebe1d083434b6789a26f"
server_key_version: "encryptionkey@0"
I20260925 13:12:52.181293 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:52.207720 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:52.210258 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:52.210408 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:52.210651 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:52.210980 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:52.211023 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.211054 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:52.211073 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.218276 9115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39245
I20260925 13:12:52.218374 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39245 every 8 connection(s)
I20260925 13:12:52.218758 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.227145 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.219485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:59601)
0925 13:12:52.219697 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.219741 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.220218 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.220759 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.220763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.221447 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.221642 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.221647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.222753 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.222754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.223569 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.223571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.224182 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.224183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.224315 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.224873 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.224882 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.227002 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260925 13:12:52.227288 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.219558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59601 (local address 127.0.98.62:35615)
0925 13:12:52.220393 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.220400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.220427 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:12:52.220524 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.220528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.220568 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.220720 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.221790 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.222636 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.223688 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.224085 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.224210 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.227126 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59601: BlockingRecv error: recv got EOF from 127.0.98.1:59601 (error 108)
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":71,"threads_started":1}
W20260925 13:12:52.227403 9263 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35615 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35615: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 13:12:52.227747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9115
I20260925 13:12:52.228094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42771
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35615
--webserver_port=35801
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.232220 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.227713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:35681)
0925 13:12:52.227790 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.227803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.227880 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.228021 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.228022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.228175 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.228233 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.228235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.228895 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.228896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.229576 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.229577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.230080 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.230081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.230130 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.230585 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.230589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.232174 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 13:12:52.232363 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.227740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35681 (local address 127.0.98.62:35615)
0925 13:12:52.227778 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.227779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.227919 (+ 140us) server_negotiation.cc:407] Connection header received
0925 13:12:52.227939 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.227940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.227949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.227997 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.228341 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.228812 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.229701 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.230003 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.230082 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.232285 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35681: BlockingRecv error: recv got EOF from 127.0.98.1:35681 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.237284 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.232666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:51985)
0925 13:12:52.232714 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.232724 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.232762 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.232903 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.232903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.233031 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.233055 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.233056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.233987 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.233988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.234451 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.234452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.234941 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.234942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.234974 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.235395 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.235400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.237197 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:12:52.237306 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.232742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51985 (local address 127.0.98.62:35615)
0925 13:12:52.232781 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.232783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.232791 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:52.232814 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.232815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.232826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.232886 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.233120 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.233912 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.234543 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.234871 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.234945 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.237267 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51985: BlockingRecv error: recv got EOF from 127.0.98.1:51985 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 13:12:52.237711 9263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:12:52.311062 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:52.311298 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:52.311338 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:52.312505 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:52.312572 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:52.312616 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:12:52.314126 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35801
--tserver_master_addrs=127.0.98.62:35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:52.314389 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:52.314615 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:52.319600 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:52.319634 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:52.319850 9267 server_base.cc:1034] running on GCE node
W20260925 13:12:52.319797 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:52.320153 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:52.320408 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:52.321548 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790341972321539 us; error 27 us; skew 500 ppm
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:52.324028 9267 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:12:52.324503 9267 webserver.cc:460] Webserver started at http://127.0.98.2:35801/ using document root <none> and password file <none>
I20260925 13:12:52.324770 9267 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:52.324862 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:52.326289 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:12:52.326956 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.327126 9267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:52.327222 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "01f8c7e4079e456db35adf3373ac78b1"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "2be26e2595528d10e300d2f188ae5da4"
server_key_iv: "c8850c0f83fe4a1fd99b3f6ba7d5c970"
server_key_version: "encryptionkey@0"
I20260925 13:12:52.327574 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:52.346196 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:52.348345 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:52.348497 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:52.348755 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:52.349081 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:52.349138 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.349180 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:52.349211 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.355044 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42771
I20260925 13:12:52.355129 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42771 every 8 connection(s)
I20260925 13:12:52.355481 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 13:12:52.362710 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.356256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45693 (local address 127.0.98.62:35615)
0925 13:12:52.356320 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.356322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.356573 (+ 251us) server_negotiation.cc:407] Connection header received
0925 13:12:52.357054 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.357055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.357069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.357121 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.358122 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.358600 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.359566 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.359742 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.359792 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.362660 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45693: BlockingRecv error: recv got EOF from 127.0.98.2:45693 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:12:52.362767 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.356259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:45693)
0925 13:12:52.356446 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.356482 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.356971 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.357204 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.357207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.357843 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.358043 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.358048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.358676 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.358677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.359492 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.359494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.359811 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.359811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.359914 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.360388 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.360396 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.362562 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
W20260925 13:12:52.363073 9396 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35615 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35615: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 13:12:52.363291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9267
I20260925 13:12:52.363626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43263
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35615
--webserver_port=41479
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.368081 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.363380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:49673)
0925 13:12:52.363449 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.363461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.363526 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.363693 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.363694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.363910 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.363971 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.363973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.364588 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.364589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.365083 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.365085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.365458 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.365459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.365530 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.366010 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.366014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.367977 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 13:12:52.368096 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.363503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49673 (local address 127.0.98.62:35615)
0925 13:12:52.363559 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.363561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.363569 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:52.363593 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.363594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.363606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.363667 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.364056 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.364512 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.365168 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.365388 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.365463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.368052 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49673: BlockingRecv error: recv got EOF from 127.0.98.2:49673 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.373044 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.368443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:42651)
0925 13:12:52.368505 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.368516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.368566 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.368717 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.368718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.368872 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.368911 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.368913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.369508 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.369509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.370430 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.370433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.370748 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.370749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.370813 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.371194 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.371200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.372993 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 13:12:52.373186 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.368509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42651 (local address 127.0.98.62:35615)
0925 13:12:52.368556 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.368558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.368573 (+ 15us) server_negotiation.cc:407] Connection header received
0925 13:12:52.368634 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.368635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.368642 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.368686 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.368985 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.369436 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.370508 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.370677 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.370727 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.373108 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42651: BlockingRecv error: recv got EOF from 127.0.98.2:42651 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 13:12:52.373215 9396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:12:52.447383 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:12:52.447623 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:12:52.447647 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:12:52.448930 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:12:52.448974 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:12:52.449002 9399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:12:52.450444 9399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41479
--tserver_master_addrs=127.0.98.62:35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:12:52.450639 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:12:52.450826 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:12:52.455961 9405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:52.455977 9404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:12:52.455961 9407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:12:52.456363 9399 server_base.cc:1034] running on GCE node
I20260925 13:12:52.456565 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:12:52.456818 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:12:52.457976 9399 hybrid_clock.cc:648] HybridClock initialized: now 1790341972457965 us; error 29 us; skew 500 ppm
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:12:52.460377 9399 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:12:52.460867 9399 webserver.cc:460] Webserver started at http://127.0.98.3:41479/ using document root <none> and password file <none>
I20260925 13:12:52.461112 9399 fs_manager.cc:359] Metadata directory not provided
I20260925 13:12:52.461198 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:12:52.462630 9399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:12:52.463238 9414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.463425 9399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:12:52.463526 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "e4fbbfd7516a4d6abbe59782349137dc"
format_stamp: "Formatted at 2026-09-25 13:12:51 on dist-test-slave-2d0s"
server_key: "a07379f56701420893fb9e2be783d4f9"
server_key_iv: "46ef20d5619cd38fd355df7d37516296"
server_key_version: "encryptionkey@0"
I20260925 13:12:52.463869 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:12:52.482402 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:12:52.484375 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:12:52.484561 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:12:52.484817 9399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:12:52.485119 9399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:12:52.485180 9399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.485244 9399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:12:52.485270 9399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:12:52.491712 9399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43263
I20260925 13:12:52.491755 9527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43263 every 8 connection(s)
I20260925 13:12:52.492101 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.499097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9399
W20260925 13:12:52.499539 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.492772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42843 (local address 127.0.98.62:35615)
0925 13:12:52.492834 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.492836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.493055 (+ 219us) server_negotiation.cc:407] Connection header received
0925 13:12:52.493538 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.493539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.493567 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.493628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.494676 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.495167 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.496202 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.496397 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.496471 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.499466 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42843: BlockingRecv error: recv got EOF from 127.0.98.3:42843 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:12:52.499563 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.492747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:42843)
0925 13:12:52.492914 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.492954 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.493453 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.493802 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.493806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.494377 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.494577 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.494583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.495249 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.495250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.496093 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.496095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.496479 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.496480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.496581 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.497116 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.497125 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.499371 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
W20260925 13:12:52.499847 9528 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35615 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35615: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.505694 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.500075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:44129)
0925 13:12:52.500128 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.500139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.500183 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.500380 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.500381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.500529 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.500585 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.500587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.501614 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.501615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.502456 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.502458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.502969 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.502970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.503035 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.503524 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.503529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.505585 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 13:12:52.505774 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.500133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44129 (local address 127.0.98.62:35615)
0925 13:12:52.500248 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.500250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.500258 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:52.500280 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.500281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.500298 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.500356 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.500670 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.501526 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.502549 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.502873 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.502951 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.505695 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44129: BlockingRecv error: recv got EOF from 127.0.98.3:44129 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260925 13:12:52.507899 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.499714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.98.62:35615)
0925 13:12:52.499754 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.499756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.499800 (+ 44us) server_negotiation.cc:407] Connection header received
0925 13:12:52.499912 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.499914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.499926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.499988 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.500790 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.501528 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.502609 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.502893 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.502957 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.504432 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:12:52.504438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:12:52.504438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:12:52.504452 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:12:52.506534 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:52.506890 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:52.506893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:52.506893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:52.506985 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:12:52.507215 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:12:52.507216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:12:52.507216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:12:52.507453 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:12:52.507566 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 13:12:52.507707 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 13:12:52.507789 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":22}
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:52 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:52.510510 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:52.506059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:49509)
0925 13:12:52.506109 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:52.506119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:52.506155 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:52.506305 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:52.506306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:52.506430 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:52.506455 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.506456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.507325 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.507326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.508087 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:52.508089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.508492 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.508493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:52.508544 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.508952 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:52.508956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:52.510466 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:12:52.510602 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:52.506135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49509 (local address 127.0.98.62:35615)
0925 13:12:52.506178 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:52.506180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:52.506188 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:12:52.506210 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:52.506211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:52.506221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:52.506302 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:52.506535 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.507240 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.508192 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:52.508427 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:52.508493 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:52.510540 (+ 2047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49509: BlockingRecv error: recv got EOF from 127.0.98.3:49509 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:12:52.510682 9528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 13:12:53 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:53 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:53.242698 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:53.238083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:46513)
0925 13:12:53.238209 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:53.238229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:53.238275 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:53.238431 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:53.238432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:53.238587 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:53.238636 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:53.238638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.239348 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.239349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:53.239908 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:53.239910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.240284 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.240285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:53.240332 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:53.240860 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:53.240864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:53.242605 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
W20260925 13:12:53.242785 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:53.238174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46513 (local address 127.0.98.62:35615)
0925 13:12:53.238300 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:53.238302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:53.238309 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:53.238330 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:53.238330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:53.238340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:53.238407 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:53.238738 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.239255 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.239996 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.240209 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.240265 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:53.242707 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46513: BlockingRecv error: recv got EOF from 127.0.98.1:46513 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
Sep 25 13:12:53 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:53 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:53.378355 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:53.373603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:47545)
0925 13:12:53.373782 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:53.373792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:53.373871 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:53.374033 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:53.374034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:53.374218 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:53.374265 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:53.374267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.374917 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.374918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:53.375485 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:53.375487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.375850 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.375851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:53.375919 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:53.376401 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:53.376405 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:53.378250 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
W20260925 13:12:53.378391 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:53.373650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47545 (local address 127.0.98.62:35615)
0925 13:12:53.373754 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:53.373756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:53.373858 (+ 102us) server_negotiation.cc:407] Connection header received
0925 13:12:53.373925 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:53.373926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:53.373954 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:53.374022 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:53.374344 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.374835 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.375572 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.375775 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.375853 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:53.378328 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47545: BlockingRecv error: recv got EOF from 127.0.98.2:47545 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 13:12:53 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:53 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:53.515775 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:53.511079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:50841)
0925 13:12:53.511221 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:53.511237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:53.511285 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:53.511462 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:53.511463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:53.511618 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:53.511666 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:53.511668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.512262 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.512263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:53.512835 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:53.512837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.513136 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.513137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:53.513175 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:53.513816 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:53.513821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:53.515658 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260925 13:12:53.515798 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:53.511269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50841 (local address 127.0.98.62:35615)
0925 13:12:53.511314 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:53.511316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:53.511330 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:12:53.511352 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:53.511352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:53.511364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:53.511434 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:53.511729 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.512198 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.512903 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:53.513076 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:53.513123 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:53.515735 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50841: BlockingRecv error: recv got EOF from 127.0.98.3:50841 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 13:12:54 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:54 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:54.247864 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:54.243221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:45155)
0925 13:12:54.243372 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:54.243384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:54.243444 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:54.243585 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:54.243586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:54.243771 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:54.243846 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:54.243848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.244558 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.244560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:54.245085 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:54.245087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.245416 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.245417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:54.245455 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:54.246024 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:54.246029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:54.247798 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
W20260925 13:12:54.247988 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:54.243318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45155 (local address 127.0.98.62:35615)
0925 13:12:54.243432 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:54.243434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:54.243446 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:12:54.243493 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:54.243493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:54.243505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:54.243572 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:54.243965 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.244468 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.245163 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.245353 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.245403 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:54.247901 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45155: BlockingRecv error: recv got EOF from 127.0.98.1:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
Sep 25 13:12:54 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:54 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:54.383332 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:54.378846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:39579)
0925 13:12:54.378964 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:54.378978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:54.379032 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:54.379171 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:54.379171 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:54.379339 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:54.379387 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:54.379388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.380008 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.380009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:54.380548 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:54.380550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.380886 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.380886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:54.380928 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:54.381464 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:54.381469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:54.383267 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260925 13:12:54.383441 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:54.378928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39579 (local address 127.0.98.62:35615)
0925 13:12:54.379012 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:54.379014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:54.379025 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:12:54.379099 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:54.379099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:54.379111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:54.379161 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:54.379442 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.379919 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.380626 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.380817 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.380873 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:54.383372 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39579: BlockingRecv error: recv got EOF from 127.0.98.2:39579 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 13:12:54 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:54 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:54.521193 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:54.516310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:52469)
0925 13:12:54.516452 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:54.516462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:54.516513 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:54.516637 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:54.516638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:54.516800 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:54.516849 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:54.516851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.517537 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.517538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:54.518219 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:54.518221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.518552 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.518553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:54.518602 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:54.519056 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:54.519060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:54.521083 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260925 13:12:54.521215 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:54.516459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52469 (local address 127.0.98.62:35615)
0925 13:12:54.516511 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:54.516515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:54.516530 (+ 15us) server_negotiation.cc:407] Connection header received
0925 13:12:54.516557 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:54.516558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:54.516570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:54.516623 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:54.516935 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.517451 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.518277 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:54.518458 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:54.518515 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:54.521155 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52469: BlockingRecv error: recv got EOF from 127.0.98.3:52469 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 13:12:55 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:55 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:55.253144 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:55.248339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:53613)
0925 13:12:55.248588 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:55.248602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:55.248683 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:55.248855 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:55.248856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:55.249023 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:55.249077 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:55.249079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.249750 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.249750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:55.250264 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:55.250265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.250583 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.250583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:55.250621 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:55.251121 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:55.251127 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:55.253089 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
W20260925 13:12:55.253304 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:55.248409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53613 (local address 127.0.98.62:35615)
0925 13:12:55.248555 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:55.248557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:55.248689 (+ 132us) server_negotiation.cc:407] Connection header received
0925 13:12:55.248758 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:55.248759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:55.248771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:55.248833 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:55.249137 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.249649 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.250341 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.250523 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.250570 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:55.253203 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53613: BlockingRecv error: recv got EOF from 127.0.98.1:53613 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
Sep 25 13:12:55 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:55 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:55.388804 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:55.383866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:56733)
0925 13:12:55.384049 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:55.384061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:55.384161 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:55.384347 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:55.384348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:55.384563 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:55.384616 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:55.384618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.385353 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.385354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:55.385943 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:55.385946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.386343 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.386344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:55.386400 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:55.387000 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:55.387005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:55.388717 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
W20260925 13:12:55.388897 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:55.383950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56733 (local address 127.0.98.62:35615)
0925 13:12:55.384029 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:55.384031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:55.384138 (+ 107us) server_negotiation.cc:407] Connection header received
0925 13:12:55.384244 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:55.384245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:55.384259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:55.384340 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:55.384726 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.385260 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.386040 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.386253 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.386343 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:55.388830 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56733: BlockingRecv error: recv got EOF from 127.0.98.2:56733 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 13:12:55 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:55 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:55.526103 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:55.521792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:51815)
0925 13:12:55.521910 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:55.521922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:55.521989 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:55.522120 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:55.522121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:55.522301 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:55.522358 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:55.522360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.523017 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.523018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:55.523541 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:55.523542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.523920 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.523921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:55.523965 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:55.524453 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:55.524459 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:55.526046 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
W20260925 13:12:55.526175 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:55.521881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51815 (local address 127.0.98.62:35615)
0925 13:12:55.521926 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:55.521928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:55.521978 (+ 50us) server_negotiation.cc:407] Connection header received
0925 13:12:55.522037 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:55.522037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:55.522050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:55.522106 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:55.522452 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.522946 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.523629 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:55.523829 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:55.523891 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:55.526129 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51815: BlockingRecv error: recv got EOF from 127.0.98.3:51815 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 13:12:56 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:56 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:56.258592 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:56.253707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:54041)
0925 13:12:56.253859 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:56.253868 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:56.253921 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:56.254143 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:56.254145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:56.254310 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:56.254360 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:56.254361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.255086 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.255086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:56.255606 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:56.255608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.255964 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.255965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:56.256001 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:56.256453 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:56.256458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:56.258536 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
W20260925 13:12:56.258760 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:56.253859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54041 (local address 127.0.98.62:35615)
0925 13:12:56.253981 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:56.253983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:56.253992 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:56.254015 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:56.254016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:56.254026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:56.254097 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:56.254447 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.255011 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.255680 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.255901 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.255952 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:56.258646 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54041: BlockingRecv error: recv got EOF from 127.0.98.1:54041 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
Sep 25 13:12:56 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:56 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:56.401592 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:56.389326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:36115)
0925 13:12:56.389496 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:56.389507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:56.389572 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:56.389807 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:56.389808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:56.389977 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:56.390027 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:56.390029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.390797 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.390798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:56.391371 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:56.391373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.391743 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.391744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:56.391787 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:56.399637 (+ 7850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:56.399642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:56.401530 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
W20260925 13:12:56.401799 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:56.389385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36115 (local address 127.0.98.62:35615)
0925 13:12:56.389461 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:56.389463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:56.389574 (+ 111us) server_negotiation.cc:407] Connection header received
0925 13:12:56.389620 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:56.389621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:56.389633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:56.389788 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:56.390137 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.390708 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.391453 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.391660 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.391722 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:56.401643 (+ 9921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36115: BlockingRecv error: recv got EOF from 127.0.98.2:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 13:12:56 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:56 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:56.531078 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:56.526610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:44999)
0925 13:12:56.526737 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:56.526750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:56.526807 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:56.526974 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:56.526975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:56.527138 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:56.527184 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:56.527186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.527793 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.527794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:56.528357 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:56.528359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.528723 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.528723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:56.528760 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:56.529185 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:56.529190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:56.531015 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260925 13:12:56.531190 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:56.526719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44999 (local address 127.0.98.62:35615)
0925 13:12:56.526786 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:56.526787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:56.526796 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:12:56.526863 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:56.526863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:56.526876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:56.526935 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:56.527255 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.527711 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.528468 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:56.528648 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:56.528716 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:56.531101 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44999: BlockingRecv error: recv got EOF from 127.0.98.3:44999 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 13:12:57 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:57 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:57.263485 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:57.259130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:53907)
0925 13:12:57.259304 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:57.259333 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:57.259394 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:57.259559 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:57.259560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:57.259734 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:57.259783 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:57.259784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.260409 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.260410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:57.260983 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:57.260984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.261313 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.261314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:57.261349 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:57.261833 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:57.261838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:57.263426 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
W20260925 13:12:57.263587 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:57.259190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53907 (local address 127.0.98.62:35615)
0925 13:12:57.259334 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:57.259336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:57.259393 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:12:57.259439 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:57.259440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:57.259449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:57.259512 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:57.259852 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.260336 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.261049 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.261256 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.261302 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:57.263509 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53907: BlockingRecv error: recv got EOF from 127.0.98.1:53907 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
Sep 25 13:12:57 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:57 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:57.406864 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:57.402079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:33599)
0925 13:12:57.402225 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:57.402237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:57.402301 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:57.402451 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:57.402451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:57.402625 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:57.402670 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:57.402672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.403339 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.403340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:57.404140 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:57.404143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.404513 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.404514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:57.404570 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:57.405000 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:57.405004 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:57.406794 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20260925 13:12:57.406970 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:57.402204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33599 (local address 127.0.98.62:35615)
0925 13:12:57.402268 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:57.402270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:57.402283 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:12:57.402358 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:57.402359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:57.402373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:57.402443 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:57.402757 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.403253 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.404230 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.404414 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.404514 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:57.406879 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33599: BlockingRecv error: recv got EOF from 127.0.98.2:33599 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 13:12:57 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:57 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:57.537441 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:57.531546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:54753)
0925 13:12:57.531678 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:57.531688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:57.531754 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:57.531931 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:57.531931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:57.532124 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:57.532182 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:57.532184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.533297 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.533299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:57.533961 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:57.533963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.534443 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.534444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:57.534503 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:57.535009 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:57.535014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:57.537338 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260925 13:12:57.537469 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:57.531626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54753 (local address 127.0.98.62:35615)
0925 13:12:57.531695 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:57.531697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:57.531746 (+ 49us) server_negotiation.cc:407] Connection header received
0925 13:12:57.531817 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:57.531818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:57.531838 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:57.531921 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:57.532308 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.533204 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.534050 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:57.534380 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:57.534437 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:57.537405 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54753: BlockingRecv error: recv got EOF from 127.0.98.3:54753 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 13:12:58 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:58 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:58.267984 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:58.263893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:38409)
0925 13:12:58.264032 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:58.264042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:58.264096 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:58.264214 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:58.264215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:58.264356 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:58.264402 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:58.264403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.264982 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.264983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:58.265462 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:58.265463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.265800 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.265801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:58.265837 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:58.266263 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:58.266273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:58.267932 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
W20260925 13:12:58.268044 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:58.263958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38409 (local address 127.0.98.62:35615)
0925 13:12:58.264052 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:58.264054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:58.264097 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:12:58.264143 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:58.264144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:58.264153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:58.264197 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:58.264456 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.264921 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.265523 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.265746 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.265793 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:58.267998 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38409: BlockingRecv error: recv got EOF from 127.0.98.1:38409 (error 108)
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":41,"threads_started":1}
Sep 25 13:12:58 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:58 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:58.411762 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:58.407279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:44949)
0925 13:12:58.407445 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:58.407455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:58.407505 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:58.407616 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:58.407617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:58.407774 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:58.407821 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:58.407823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.408488 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.408489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:58.409034 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:58.409035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.409426 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.409428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:58.409486 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:58.409999 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:58.410004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:58.411666 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
W20260925 13:12:58.411809 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:58.407372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44949 (local address 127.0.98.62:35615)
0925 13:12:58.407431 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:58.407432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:58.407492 (+ 60us) server_negotiation.cc:407] Connection header received
0925 13:12:58.407542 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:58.407542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:58.407551 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:58.407597 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:58.407904 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.408386 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.409118 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.409319 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.409413 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:58.411747 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44949: BlockingRecv error: recv got EOF from 127.0.98.2:44949 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 13:12:58 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:58 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:58.542666 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:58.537983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:42229)
0925 13:12:58.538157 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:58.538168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:58.538257 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:58.538384 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:58.538384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:58.538554 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:58.538598 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:58.538600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.539263 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.539264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:58.539816 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:58.539817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.540186 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.540187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:58.540257 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:58.540736 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:58.540741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:58.542606 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
W20260925 13:12:58.542783 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:58.538069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42229 (local address 127.0.98.62:35615)
0925 13:12:58.538139 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:58.538141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:58.538238 (+ 97us) server_negotiation.cc:407] Connection header received
0925 13:12:58.538315 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:58.538316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:58.538330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:58.538379 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:58.538679 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.539180 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.539910 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:58.540101 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:58.540185 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:58.542696 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42229: BlockingRecv error: recv got EOF from 127.0.98.3:42229 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 13:12:59 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:59 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:59.273005 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:59.268443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:46871)
0925 13:12:59.268604 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:59.268616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:59.268687 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:59.268831 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:59.268832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:59.268997 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:59.269068 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:59.269070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.269739 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.269740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:59.270293 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:59.270295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.270635 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.270636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:59.270685 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:59.271144 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:59.271148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:59.272946 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
W20260925 13:12:59.273154 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:59.268561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46871 (local address 127.0.98.62:35615)
0925 13:12:59.268698 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:59.268700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:59.268707 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:12:59.268747 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:59.268748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:59.268759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:59.268810 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:59.269131 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.269638 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.270367 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.270555 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.270605 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:59.273056 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46871: BlockingRecv error: recv got EOF from 127.0.98.1:46871 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
Sep 25 13:12:59 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:59 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:59.416690 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:59.412274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:53455)
0925 13:12:59.412423 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:59.412434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:59.412480 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:59.412610 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:59.412611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:59.412790 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:59.412839 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:59.412840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.413487 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.413488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:59.414063 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:59.414065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.414391 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.414391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:59.414432 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:59.414893 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:59.414897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:59.416633 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260925 13:12:59.416800 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:59.412397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53455 (local address 127.0.98.62:35615)
0925 13:12:59.412459 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:59.412461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:59.412471 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:12:59.412540 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:59.412541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:59.412551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:59.412594 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:59.412930 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.413416 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.414135 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.414325 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.414378 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:59.416720 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53455: BlockingRecv error: recv got EOF from 127.0.98.2:53455 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 13:12:59 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:12:59 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:12:59.548022 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:12:59.543190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:55739)
0925 13:12:59.543353 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:12:59.543364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:12:59.543415 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:12:59.543548 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:12:59.543549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:12:59.543713 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:12:59.543795 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:59.543798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.544407 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.544408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:59.544925 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:12:59.544927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.545304 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.545305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:12:59.545354 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:59.545931 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:12:59.545935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:12:59.547945 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260925 13:12:59.548143 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:12:59.543293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55739 (local address 127.0.98.62:35615)
0925 13:12:59.543351 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:12:59.543352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:12:59.543417 (+ 65us) server_negotiation.cc:407] Connection header received
0925 13:12:59.543457 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:12:59.543458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:12:59.543469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:12:59.543527 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:12:59.543856 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.544343 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.544997 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:12:59.545194 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:12:59.545253 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:12:59.548041 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55739: BlockingRecv error: recv got EOF from 127.0.98.3:55739 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 13:13:00 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:00 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:00.277830 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:00.273488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:46215)
0925 13:13:00.273612 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:00.273627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:00.273710 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:00.273929 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:00.273930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:00.274096 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:00.274142 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:00.274143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.274758 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.274759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:00.275270 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:00.275271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.275584 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.275585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:00.275622 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:00.276060 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:00.276064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:00.277773 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260925 13:13:00.277976 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:00.273654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46215 (local address 127.0.98.62:35615)
0925 13:13:00.273812 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:00.273814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:00.273821 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:13:00.273841 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:00.273841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:00.273851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:00.273910 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:00.274206 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.274688 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.275341 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.275525 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.275572 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:00.277883 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46215: BlockingRecv error: recv got EOF from 127.0.98.1:46215 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
Sep 25 13:13:00 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:00 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:00.422202 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:00.417324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:42903)
0925 13:13:00.417529 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:00.417540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:00.417603 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:00.417762 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:00.417762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:00.417929 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:00.417978 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:00.417980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.418770 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.418771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:00.419311 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:00.419312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.419699 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.419700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:00.419774 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:00.420236 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:00.420240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:00.422143 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":122,"threads_started":1}
W20260925 13:13:00.422369 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:00.417384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42903 (local address 127.0.98.62:35615)
0925 13:13:00.417461 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:00.417464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:00.417606 (+ 142us) server_negotiation.cc:407] Connection header received
0925 13:13:00.417648 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:00.417649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:00.417680 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:00.417743 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:00.418043 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.418685 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.419406 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.419602 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.419666 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:00.422285 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42903: BlockingRecv error: recv got EOF from 127.0.98.2:42903 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 13:13:00 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:00 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:00.553258 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:00.548471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:54759)
0925 13:13:00.548632 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:00.548646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:00.548727 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:00.548895 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:00.548896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:00.549070 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:00.549118 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:00.549120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.549856 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.549857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:00.550391 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:00.550393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.550761 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.550762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:00.550829 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:00.551351 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:00.551356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:00.553165 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
W20260925 13:13:00.553316 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:00.548606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54759 (local address 127.0.98.62:35615)
0925 13:13:00.548696 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:00.548697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:00.548708 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:13:00.548810 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:00.548811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:00.548824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:00.548878 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:00.549199 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.549774 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.550479 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:00.550683 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:00.550762 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:00.553252 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54759: BlockingRecv error: recv got EOF from 127.0.98.3:54759 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 13:13:01 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:01 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:01.282879 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:01.278303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:47753)
0925 13:13:01.278453 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:01.278463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:01.278515 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:01.278642 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:01.278643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:01.278815 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:01.278868 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:01.278869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.279527 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.279528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:01.280069 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:01.280070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.280414 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.280415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:01.280469 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:01.281010 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:01.281014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:01.282820 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
W20260925 13:13:01.283011 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:01.278401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47753 (local address 127.0.98.62:35615)
0925 13:13:01.278498 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:01.278500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:01.278508 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:13:01.278559 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:01.278560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:01.278570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:01.278625 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:01.278932 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.279453 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.280142 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.280333 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.280400 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:01.282908 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47753: BlockingRecv error: recv got EOF from 127.0.98.1:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":45,"threads_started":1}
Sep 25 13:13:01 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:01 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:01.430987 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:01.422768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:36881)
0925 13:13:01.422893 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:01.422910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:01.422960 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:01.423127 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:01.423128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:01.423321 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:01.423369 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:01.423371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.424035 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.424036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:01.424581 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:01.424582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.424889 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.424890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:01.424927 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:01.428791 (+ 3864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:01.428795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:01.430899 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260925 13:13:01.431075 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:01.422868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36881 (local address 127.0.98.62:35615)
0925 13:13:01.422954 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:01.422956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:01.422966 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:13:01.423017 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:01.423017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:01.423027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:01.423079 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:01.423447 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.423954 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.424650 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.424829 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.424878 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:01.431006 (+ 6128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36881: BlockingRecv error: recv got EOF from 127.0.98.2:36881 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 13:13:01 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:01 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:01.558363 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:01.553793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:48443)
0925 13:13:01.553951 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:01.553961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:01.554017 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:01.554151 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:01.554151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:01.554315 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:01.554362 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:01.554364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.554974 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.554975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:01.555507 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:01.555508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.555876 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.555877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:01.555922 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:01.556541 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:01.556545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:01.558306 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
W20260925 13:13:01.558470 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:01.553854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48443 (local address 127.0.98.62:35615)
0925 13:13:01.553919 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:01.553921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:01.554018 (+ 97us) server_negotiation.cc:407] Connection header received
0925 13:13:01.554074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:01.554075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:01.554088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:01.554128 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:01.554423 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.554890 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.555612 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:01.555797 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:01.555858 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:01.558398 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48443: BlockingRecv error: recv got EOF from 127.0.98.3:48443 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 13:13:02 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:02 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:02.287940 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:02.283436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:37869)
0925 13:13:02.283570 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:02.283581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:02.283631 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:02.283742 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:02.283743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:02.283904 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:02.283954 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:02.283955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.284591 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.284592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:02.285128 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:02.285129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.285563 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.285564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:02.285606 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:02.286135 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:02.286140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:02.287852 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260925 13:13:02.287994 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:02.283520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37869 (local address 127.0.98.62:35615)
0925 13:13:02.283643 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:02.283645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:02.283652 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:13:02.283673 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:02.283673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:02.283683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:02.283738 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:02.284013 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.284519 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.285220 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.285497 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.285561 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:02.287942 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37869: BlockingRecv error: recv got EOF from 127.0.98.1:37869 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
Sep 25 13:13:02 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:02 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:02.436090 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:02.431650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:33597)
0925 13:13:02.431769 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:02.431786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:02.431847 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:02.431956 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:02.431957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:02.432124 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:02.432183 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:02.432184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.432815 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.432817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:02.433374 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:02.433376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.433752 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.433753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:02.433801 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:02.434234 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:02.434238 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:02.436032 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260925 13:13:02.436233 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:02.431708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33597 (local address 127.0.98.62:35615)
0925 13:13:02.431771 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:02.431773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:02.431835 (+ 62us) server_negotiation.cc:407] Connection header received
0925 13:13:02.431892 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:02.431892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:02.431903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:02.431950 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:02.432240 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.432724 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.433452 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.433632 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.433753 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:02.436138 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33597: BlockingRecv error: recv got EOF from 127.0.98.2:33597 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 13:13:02 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:02 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:02.563436 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:02.558949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:38049)
0925 13:13:02.559085 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:02.559096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:02.559146 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:02.559308 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:02.559309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:02.559463 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:02.559511 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:02.559513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.560188 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.560189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:02.560709 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:02.560711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.561126 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.561128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:02.561191 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:02.561629 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:02.561633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:02.563341 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260925 13:13:02.563478 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:02.559089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38049 (local address 127.0.98.62:35615)
0925 13:13:02.559139 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:02.559141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:02.559155 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:13:02.559188 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:02.559189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:02.559201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:02.559268 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:02.559601 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.560116 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.560840 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:02.561044 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:02.561126 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:02.563416 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38049: BlockingRecv error: recv got EOF from 127.0.98.3:38049 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 13:13:03 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:03 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:03.293165 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:03.288444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:34355)
0925 13:13:03.288591 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:03.288602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:03.288660 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:03.288784 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:03.288785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:03.288946 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:03.288997 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:03.288999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.289613 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.289614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:03.290262 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:03.290265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.290586 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.290586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:03.290626 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:03.291189 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:03.291194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:03.293069 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260925 13:13:03.293221 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:03.288539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34355 (local address 127.0.98.62:35615)
0925 13:13:03.288663 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:03.288665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:03.288691 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:13:03.288711 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:03.288711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:03.288721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:03.288769 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:03.289060 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.289546 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.290335 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.290521 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.290573 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:03.293148 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34355: BlockingRecv error: recv got EOF from 127.0.98.1:34355 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
Sep 25 13:13:03 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:03 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:03.441162 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:03.436637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:58743)
0925 13:13:03.436826 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:03.436846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:03.436904 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:03.437047 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:03.437048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:03.437209 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:03.437258 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:03.437259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.437917 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.437917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:03.438461 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:03.438463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.438794 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.438795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:03.438832 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:03.439290 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:03.439294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:03.441105 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
W20260925 13:13:03.441339 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:03.436775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58743 (local address 127.0.98.62:35615)
0925 13:13:03.436871 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:03.436873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:03.436887 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:13:03.436968 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:03.436969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:03.436980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:03.437023 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:03.437322 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.437855 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.438542 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.438730 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.438783 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:03.441225 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58743: BlockingRecv error: recv got EOF from 127.0.98.2:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 13:13:03 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:03 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:03.568359 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:03.563914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:55577)
0925 13:13:03.564040 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:03.564050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:03.564100 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:03.564244 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:03.564245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:03.564410 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:03.564459 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:03.564460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.565103 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.565104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:03.565614 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:03.565616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.565995 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.565996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:03.566037 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:03.566535 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:03.566540 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:03.568301 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
W20260925 13:13:03.568467 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:03.563976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55577 (local address 127.0.98.62:35615)
0925 13:13:03.564078 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:03.564079 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:03.564089 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:13:03.564145 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:03.564145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:03.564158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:03.564229 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:03.564526 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.565027 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.565742 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:03.565929 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:03.565989 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:03.568393 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55577: BlockingRecv error: recv got EOF from 127.0.98.3:55577 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 13:13:04 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:04 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:04.298403 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:04.293729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:43091)
0925 13:13:04.293914 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:04.293925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:04.293987 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:04.294129 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:04.294130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:04.294292 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:04.294343 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:04.294344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.295037 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.295037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:04.295543 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:04.295545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.295944 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.295945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:04.295987 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:04.296493 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:04.296498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:04.298332 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":121,"threads_started":1}
W20260925 13:13:04.298600 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:04.293749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43091 (local address 127.0.98.62:35615)
0925 13:13:04.293915 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:04.293918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:04.293971 (+ 53us) server_negotiation.cc:407] Connection header received
0925 13:13:04.294044 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:04.294044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:04.294057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:04.294114 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:04.294413 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.294966 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.295655 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.295876 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.295936 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:04.298494 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43091: BlockingRecv error: recv got EOF from 127.0.98.1:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
Sep 25 13:13:04 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:04 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:04.446187 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:04.441767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:58133)
0925 13:13:04.441899 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:04.441911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:04.441977 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:04.442100 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:04.442101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:04.442257 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:04.442303 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:04.442305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.442963 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.442964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:04.443455 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:04.443457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.443760 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.443761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:04.443796 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:04.444252 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:04.444256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:04.446128 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260925 13:13:04.446301 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:04.441800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58133 (local address 127.0.98.62:35615)
0925 13:13:04.441854 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:04.441856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:04.441969 (+ 113us) server_negotiation.cc:407] Connection header received
0925 13:13:04.442020 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:04.442021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:04.442032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:04.442093 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:04.442357 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.442886 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.443523 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.443700 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.443752 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:04.446208 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58133: BlockingRecv error: recv got EOF from 127.0.98.2:58133 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 13:13:04 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:04 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:04.573951 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:04.568872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:44635)
0925 13:13:04.569080 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:04.569097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:04.569186 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:04.569354 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:04.569354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:04.569536 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:04.569585 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:04.569587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.570319 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.570320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:04.570854 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:04.570855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.571216 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.571217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:04.571285 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:04.571888 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:04.571894 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:04.573888 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
W20260925 13:13:04.574060 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:04.568952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44635 (local address 127.0.98.62:35615)
0925 13:13:04.569041 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:04.569043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:04.569165 (+ 122us) server_negotiation.cc:407] Connection header received
0925 13:13:04.569261 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:04.569262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:04.569275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:04.569348 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:04.569692 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.570227 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.570947 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:04.571132 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:04.571205 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:04.573988 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44635: BlockingRecv error: recv got EOF from 127.0.98.3:44635 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 13:13:05 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:05 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:05.303570 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:05.298864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:35509)
0925 13:13:05.298999 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:05.299010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:05.299067 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:05.299254 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:05.299255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:05.299411 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:05.299458 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:05.299459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.300151 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.300152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:05.300681 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:05.300682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.301015 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.301017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:05.301063 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:05.301554 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:05.301558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:05.303512 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260925 13:13:05.303720 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:05.298954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35509 (local address 127.0.98.62:35615)
0925 13:13:05.299087 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:05.299088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:05.299097 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:13:05.299119 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:05.299120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:05.299130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:05.299213 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:05.299557 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.300042 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.300755 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.300944 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.301002 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:05.303620 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35509: BlockingRecv error: recv got EOF from 127.0.98.1:35509 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
Sep 25 13:13:05 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:05 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:05.451574 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:05.446875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:53143)
0925 13:13:05.447024 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:05.447036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:05.447104 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:05.447323 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:05.447324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:05.447533 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:05.447592 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:05.447594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.448367 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.448368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:05.448870 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:05.448871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.449181 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.449182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:05.449220 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:05.449755 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:05.449760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:05.451491 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
W20260925 13:13:05.451753 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:05.446994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53143 (local address 127.0.98.62:35615)
0925 13:13:05.447052 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:05.447054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:05.447089 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:13:05.447163 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:05.447164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:05.447189 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:05.447273 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:05.447672 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.448293 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.448941 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.449117 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.449168 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:05.451637 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53143: BlockingRecv error: recv got EOF from 127.0.98.2:53143 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 13:13:05 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:05 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:05.579150 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:05.574525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:60549)
0925 13:13:05.574652 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:05.574671 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:05.574743 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:05.574874 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:05.574874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:05.575035 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:05.575084 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:05.575086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.575720 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.575721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:05.576272 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:05.576274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.576593 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.576594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:05.576632 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:05.577215 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:05.577220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:05.579036 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260925 13:13:05.579210 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:05.574612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60549 (local address 127.0.98.62:35615)
0925 13:13:05.574672 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:05.574674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:05.574740 (+ 66us) server_negotiation.cc:407] Connection header received
0925 13:13:05.574809 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:05.574810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:05.574823 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:05.574861 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:05.575141 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.575647 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.576348 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:05.576529 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:05.576581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:05.579140 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60549: BlockingRecv error: recv got EOF from 127.0.98.3:60549 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 13:13:06 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:06 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:06.308580 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:06.304061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:56429)
0925 13:13:06.304200 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:06.304210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:06.304256 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:06.304394 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:06.304395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:06.304554 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:06.304602 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:06.304603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.305238 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.305240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:06.305834 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:06.305836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.306143 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.306144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:06.306183 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:06.306690 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:06.306695 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:06.308523 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
W20260925 13:13:06.308718 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:06.304141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56429 (local address 127.0.98.62:35615)
0925 13:13:06.304265 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:06.304267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:06.304276 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:13:06.304300 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:06.304300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:06.304320 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:06.304379 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:06.304666 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.305173 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.305906 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.306086 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.306134 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:06.308624 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56429: BlockingRecv error: recv got EOF from 127.0.98.1:56429 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
Sep 25 13:13:06 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:06 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:06.460462 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:06.452114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:39239)
0925 13:13:06.452276 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:06.452288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:06.452361 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:06.452495 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:06.452496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:06.452656 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:06.452705 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:06.452706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.453374 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.453375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:06.453928 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:06.453930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.454283 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.454284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:06.454324 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:06.458533 (+ 4209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:06.458538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:06.460403 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
W20260925 13:13:06.460573 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:06.452213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39239 (local address 127.0.98.62:35615)
0925 13:13:06.452291 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:06.452292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:06.452345 (+ 53us) server_negotiation.cc:407] Connection header received
0925 13:13:06.452421 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:06.452422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:06.452433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:06.452474 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:06.452767 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.453299 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.454021 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.454216 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.454272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:06.460499 (+ 6227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39239: BlockingRecv error: recv got EOF from 127.0.98.2:39239 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 13:13:06 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:06 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:06.584789 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:06.579801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.3:56093)
0925 13:13:06.579967 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:06.579979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:06.580074 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:06.580286 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:06.580288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:06.580484 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:06.580533 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:06.580535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.581241 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.581243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:06.581895 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:06.581897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.582261 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.582262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:06.582322 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:06.582886 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:06.582890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:06.584719 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
W20260925 13:13:06.584913 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:06.579870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56093 (local address 127.0.98.62:35615)
0925 13:13:06.579933 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:06.579935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:06.580052 (+ 117us) server_negotiation.cc:407] Connection header received
0925 13:13:06.580139 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:06.580140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:06.580155 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:06.580272 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:06.580619 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.581143 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.581981 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:06.582184 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:06.582264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:06.584801 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56093: BlockingRecv error: recv got EOF from 127.0.98.3:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 13:13:07 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:07 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:07.314011 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.309235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.1:48423)
0925 13:13:07.309393 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:07.309413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:07.309469 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:07.309611 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:07.309612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:07.309833 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:07.309887 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.309889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.310549 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.310550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.311107 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.311109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.311474 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.311475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.311523 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.312151 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:07.312157 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:07.313942 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
W20260925 13:13:07.314083 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:07.309298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48423 (local address 127.0.98.62:35615)
0925 13:13:07.309436 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:07.309438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:07.309452 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:13:07.309520 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:07.309520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:07.309530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:07.309594 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:07.309964 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.310471 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.311190 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.311384 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.311447 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.314032 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48423: BlockingRecv error: recv got EOF from 127.0.98.1:48423 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
Sep 25 13:13:07 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:13:07 dist-test-slave-2d0s krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:13:07.466008 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.461114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35615 (local address 127.0.98.2:45425)
0925 13:13:07.461259 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:07.461276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:07.461343 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:07.461509 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:07.461510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:07.461694 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:07.461743 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.461744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.462409 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.462410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.463215 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.463218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.463637 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.463638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.463678 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.464196 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:07.464202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:07.465951 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35615: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
W20260925 13:13:07.466141 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:07.461186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45425 (local address 127.0.98.62:35615)
0925 13:13:07.461302 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:07.461303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:07.461333 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:13:07.461413 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:07.461414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:07.461427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:07.461495 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:07.461835 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.462342 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.463309 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.463571 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.463629 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.466064 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45425: BlockingRecv error: recv got EOF from 127.0.98.2:45425 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 13:13:07.506181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9115
I20260925 13:13:07.510743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9267
I20260925 13:13:07.515134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9399
I20260925 13:13:07.519379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9045
2026-09-25T13:13:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17397 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T13:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:13:07Z Disabled control of system clock
I20260925 13:13:07.546988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37689
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 13:13:07.635133 9595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:07.635350 9595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:07.635370 9595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:07.636467 9595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:13:07.636502 9595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:07.636513 9595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:07.636525 9595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:13:07.636536 9595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:13:07.637799 9595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:07.637980 9595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:07.638191 9595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:07.643128 9603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:07.643129 9601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:07.643198 9600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:07.643297 9595 server_base.cc:1034] running on GCE node
I20260925 13:13:07.643667 9595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:07.644019 9595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:07.645177 9595 hybrid_clock.cc:648] HybridClock initialized: now 1790341987645158 us; error 37 us; skew 500 ppm
I20260925 13:13:07.646516 9595 webserver.cc:460] Webserver started at http://127.0.98.62:41659/ using document root <none> and password file <none>
I20260925 13:13:07.646773 9595 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:07.646849 9595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:07.646987 9595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:07.648101 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "b7b7ab1e945d45d59a107766c5580542"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "3ae94a34c0140303cfb6eb2a4386085c"
server_key_iv: "fd4847b16123f2bb1dfdb64d3df4f344"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.648479 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7b7ab1e945d45d59a107766c5580542"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "3ae94a34c0140303cfb6eb2a4386085c"
server_key_iv: "fd4847b16123f2bb1dfdb64d3df4f344"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.649859 9595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:13:07.650579 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.650753 9595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:13:07.650864 9595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "b7b7ab1e945d45d59a107766c5580542"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "3ae94a34c0140303cfb6eb2a4386085c"
server_key_iv: "fd4847b16123f2bb1dfdb64d3df4f344"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.650938 9595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:07.668727 9595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:07.669116 9595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:07.669277 9595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:07.673298 9595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37689
I20260925 13:13:07.673368 9661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37689 every 8 connection(s)
I20260925 13:13:07.673794 9595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:13:07.674877 9662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:07.677266 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542: Bootstrap starting.
I20260925 13:13:07.677975 9662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:07.678283 9662 log.cc:824] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:07.678992 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542: No bootstrap required, opened a new log
I20260925 13:13:07.680713 9662 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b7ab1e945d45d59a107766c5580542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260925 13:13:07.680857 9662 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:07.680881 9662 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b7ab1e945d45d59a107766c5580542, State: Initialized, Role: FOLLOWER
I20260925 13:13:07.681021 9662 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b7ab1e945d45d59a107766c5580542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260925 13:13:07.681107 9662 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:13:07.681131 9662 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:13:07.681159 9662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:07.681712 9662 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b7ab1e945d45d59a107766c5580542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260925 13:13:07.681816 9662 leader_election.cc:302] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b7ab1e945d45d59a107766c5580542; no voters:
I20260925 13:13:07.681972 9662 leader_election.cc:288] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:13:07.682006 9665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:13:07.682116 9665 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [term 1 LEADER]: Becoming Leader. State: Replica: b7b7ab1e945d45d59a107766c5580542, State: Running, Role: LEADER
I20260925 13:13:07.682232 9662 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:13:07.682268 9665 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b7ab1e945d45d59a107766c5580542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260925 13:13:07.682464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9595
I20260925 13:13:07.682569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:13:07.682777 392 external_mini_cluster.cc:1428] Setting key 10c3601eea3e2929e59cc10069ac2276
I20260925 13:13:07.683151 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b7ab1e945d45d59a107766c5580542. Latest consensus state: current_term: 1 leader_uuid: "b7b7ab1e945d45d59a107766c5580542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b7ab1e945d45d59a107766c5580542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } } }
I20260925 13:13:07.683290 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:07.683569 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b7ab1e945d45d59a107766c5580542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b7ab1e945d45d59a107766c5580542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } } }
I20260925 13:13:07.683642 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542 [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:07.684062 9678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:13:07.684777 9678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:13:07.686455 9678 catalog_manager.cc:1347] Generated new cluster ID: a637ff9ebb684fabbc4f123c67d8aee2
I20260925 13:13:07.686510 9678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:13:07.689898 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.684103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.98.62:37689)
0925 13:13:07.684300 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:07.684307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:07.684342 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:13:07.684451 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:07.684457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:07.684512 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:07.684680 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:07.684958 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.686139 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.687288 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.687949 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.688450 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.688909 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:07.688913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:07.688913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:07.689128 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:07.689168 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:07.689182 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:07.689499 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:07.689534 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 13:13:07.689617 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260925 13:13:07.691715 9678 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:13:07.692421 9678 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:13:07.696391 9678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b7ab1e945d45d59a107766c5580542: Generated new TSK 0
I20260925 13:13:07.696584 9678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:13:07.702958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--builtin_ntp_servers=127.0.98.20:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 13:13:07.788033 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:07.788233 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:07.788256 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:07.789376 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:07.789413 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:07.789438 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:13:07.790841 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:07.791045 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:07.791393 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:07.796588 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:07.796556 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:07.796634 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:07.797108 9686 server_base.cc:1034] running on GCE node
I20260925 13:13:07.797349 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:07.797644 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:07.798835 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790341987798811 us; error 42 us; skew 500 ppm
I20260925 13:13:07.800138 9686 webserver.cc:460] Webserver started at http://127.0.98.1:41581/ using document root <none> and password file <none>
I20260925 13:13:07.800421 9686 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:07.800496 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:07.800652 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:07.801823 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "812f39ae4f2f4dfe9adadcab07c48b76"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "ced7850a9208528359009f7280366fb3"
server_key_iv: "e9f9a8b70681bc09605a4d8713ea76a2"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.802201 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "812f39ae4f2f4dfe9adadcab07c48b76"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "ced7850a9208528359009f7280366fb3"
server_key_iv: "e9f9a8b70681bc09605a4d8713ea76a2"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.803579 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:13:07.804399 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.804611 9686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:13:07.804715 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "812f39ae4f2f4dfe9adadcab07c48b76"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "ced7850a9208528359009f7280366fb3"
server_key_iv: "e9f9a8b70681bc09605a4d8713ea76a2"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.804797 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:07.828078 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:07.828490 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:07.828655 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:07.828917 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:07.829293 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:07.829350 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.829386 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:07.829416 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.835166 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39221
I20260925 13:13:07.835250 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39221 every 8 connection(s)
I20260925 13:13:07.835639 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:13:07.838423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9686
I20260925 13:13:07.838526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:13:07.838753 392 external_mini_cluster.cc:1428] Setting key e4fdaf20b82278a9732ab558aa1c4599
I20260925 13:13:07.839807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--builtin_ntp_servers=127.0.98.20:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 13:13:07.841737 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.836449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36451 (local address 127.0.98.62:37689)
0925 13:13:07.836516 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:07.836518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:07.836750 (+ 232us) server_negotiation.cc:407] Connection header received
0925 13:13:07.837256 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:07.837257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:07.837269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:07.837323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:07.837988 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.838505 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.840033 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.840259 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.840317 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.841347 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:07.841349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:07.841349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:07.841354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:07.841384 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:07.841389 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:07.841449 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:07.841541 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 13:13:07.841581 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:13:07.841873 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.836450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37689 (local address 127.0.98.1:36451)
0925 13:13:07.836618 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:07.836652 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:07.837159 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:07.837380 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:07.837383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:07.837392 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:07.837915 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.837920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.838621 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.838623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.839912 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.839916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.840354 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.840356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.840528 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.841179 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:13:07.841179 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:07.841220 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:13:07.841255 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:07.841456 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:07.841460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:07.841463 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:13:07.841516 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 13:13:07.841603 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260925 13:13:07.842309 9814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37689
I20260925 13:13:07.842432 9814 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:07.842634 9814 heartbeater.cc:511] Master 127.0.98.62:37689 requested a full tablet report, sending...
I20260925 13:13:07.843155 9626 ts_manager.cc:194] Registered new tserver with Master: 812f39ae4f2f4dfe9adadcab07c48b76 (127.0.98.1:39221)
I20260925 13:13:07.843887 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36451
W20260925 13:13:07.923617 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:07.923889 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:07.923924 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:07.925107 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:07.925174 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:07.925215 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:13:07.926616 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:07.926915 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:07.927223 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:07.932382 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:07.932425 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:07.932569 9817 server_base.cc:1034] running on GCE node
W20260925 13:13:07.932381 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:07.932891 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:07.933132 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:07.934270 9817 hybrid_clock.cc:648] HybridClock initialized: now 1790341987934248 us; error 44 us; skew 500 ppm
I20260925 13:13:07.935680 9817 webserver.cc:460] Webserver started at http://127.0.98.2:33701/ using document root <none> and password file <none>
I20260925 13:13:07.935940 9817 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:07.935995 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:07.936178 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:07.937284 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "7000298285744936ac6d1ebd351752c9"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "b3bf8495c1e85b907473716860444e1f"
server_key_iv: "f18bb7515a56210ac13f2e2f94c0ee1a"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.937701 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7000298285744936ac6d1ebd351752c9"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "b3bf8495c1e85b907473716860444e1f"
server_key_iv: "f18bb7515a56210ac13f2e2f94c0ee1a"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.939119 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:13:07.939867 9831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.940052 9817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:13:07.940137 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "7000298285744936ac6d1ebd351752c9"
format_stamp: "Formatted at 2026-09-25 13:13:07 on dist-test-slave-2d0s"
server_key: "b3bf8495c1e85b907473716860444e1f"
server_key_iv: "f18bb7515a56210ac13f2e2f94c0ee1a"
server_key_version: "encryptionkey@0"
I20260925 13:13:07.940189 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:07.955691 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:07.955989 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:07.956096 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:07.956290 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:07.956650 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:07.956691 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.956738 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:07.956763 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:07.962504 9817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35107
I20260925 13:13:07.962589 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35107 every 8 connection(s)
I20260925 13:13:07.962893 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:13:07.965553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9817
I20260925 13:13:07.965682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:13:07.965906 392 external_mini_cluster.cc:1428] Setting key 9995aebfebc271ba5e595b424a6e6435
I20260925 13:13:07.966852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--builtin_ntp_servers=127.0.98.20:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 13:13:07.968672 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.963694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57531 (local address 127.0.98.62:37689)
0925 13:13:07.963757 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:07.963759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:07.964031 (+ 272us) server_negotiation.cc:407] Connection header received
0925 13:13:07.964579 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:07.964581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:07.964593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:07.964646 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:07.965412 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.966068 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.967236 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.967451 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.967527 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.968408 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:07.968409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:07.968409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:07.968415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:07.968430 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:07.968434 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:07.968483 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:07.968562 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 13:13:07.968592 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:13:07.968945 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:07.963684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37689 (local address 127.0.98.2:57531)
0925 13:13:07.963879 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:07.963937 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:07.964469 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:07.964725 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:07.964728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:07.964737 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:07.965252 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.965257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.966160 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.966160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.967151 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:07.967154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:07.967532 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:07.967533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:07.967679 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:07.968256 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:13:07.968257 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:07.968289 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:13:07.968313 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:07.968482 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:07.968487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:07.968490 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:13:07.968564 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:13:07.968685 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260925 13:13:07.969445 9945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37689
I20260925 13:13:07.969547 9945 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:07.969767 9945 heartbeater.cc:511] Master 127.0.98.62:37689 requested a full tablet report, sending...
I20260925 13:13:07.970152 9626 ts_manager.cc:194] Registered new tserver with Master: 7000298285744936ac6d1ebd351752c9 (127.0.98.2:35107)
I20260925 13:13:07.970616 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57531
W20260925 13:13:08.053357 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:08.053608 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:08.053647 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:08.054860 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:08.054939 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:08.054983 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:13:08.056358 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:08.056627 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:08.056942 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:08.061941 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.061928 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.061944 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:08.062269 9948 server_base.cc:1034] running on GCE node
I20260925 13:13:08.062502 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:08.062733 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:08.063884 9948 hybrid_clock.cc:648] HybridClock initialized: now 1790341988063868 us; error 38 us; skew 500 ppm
I20260925 13:13:08.065147 9948 webserver.cc:460] Webserver started at http://127.0.98.3:44153/ using document root <none> and password file <none>
I20260925 13:13:08.065412 9948 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:08.065459 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:08.065608 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:08.066723 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7a221f6cfb942fdbc0127475bb72c6e"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "cf680c0de9a02e9272513d6eab7650dc"
server_key_iv: "4624f7d3bc9350fe209c8ba6520a82c2"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.067099 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7a221f6cfb942fdbc0127475bb72c6e"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "cf680c0de9a02e9272513d6eab7650dc"
server_key_iv: "4624f7d3bc9350fe209c8ba6520a82c2"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.068419 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:13:08.069155 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.069346 9948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:13:08.069425 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "d7a221f6cfb942fdbc0127475bb72c6e"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "cf680c0de9a02e9272513d6eab7650dc"
server_key_iv: "4624f7d3bc9350fe209c8ba6520a82c2"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.069481 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:08.098065 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:08.098354 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:08.098456 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:08.098649 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:08.099027 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:08.099064 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.099087 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:08.099138 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.104875 9948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38127
I20260925 13:13:08.104971 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38127 every 8 connection(s)
I20260925 13:13:08.105296 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:13:08.110600 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.106114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57425 (local address 127.0.98.62:37689)
0925 13:13:08.106180 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.106182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.106409 (+ 227us) server_negotiation.cc:407] Connection header received
0925 13:13:08.106890 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.106891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.106901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.106969 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.107659 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:08.108202 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:08.109241 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:08.109436 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:08.109494 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:08.110360 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:08.110361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:08.110361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:08.110366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:08.110381 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:08.110384 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:08.110429 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:08.110515 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 13:13:08.110539 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:13:08.110822 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.106073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37689 (local address 127.0.98.3:57425)
0925 13:13:08.106266 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:08.106308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:08.106810 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:08.107030 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:08.107034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:08.107043 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:08.107559 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:08.107563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:08.108281 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:08.108281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:08.109172 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:08.109174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:08.109512 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:08.109513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:08.109626 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:08.110218 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:13:08.110219 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:08.110249 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:13:08.110272 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:08.110441 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:08.110445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:08.110448 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:13:08.110501 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 13:13:08.110583 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260925 13:13:08.111379 10076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37689
I20260925 13:13:08.111490 10076 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:08.111686 10076 heartbeater.cc:511] Master 127.0.98.62:37689 requested a full tablet report, sending...
I20260925 13:13:08.112085 9626 ts_manager.cc:194] Registered new tserver with Master: d7a221f6cfb942fdbc0127475bb72c6e (127.0.98.3:38127)
I20260925 13:13:08.112422 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57425
I20260925 13:13:08.112581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 9948
I20260925 13:13:08.112679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:13:08.112911 392 external_mini_cluster.cc:1428] Setting key e5422627c38a04b8587b1744815c7af6
I20260925 13:13:08.113876 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 13:13:08.115527 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:08.115141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.98.62:37689)
0925 13:13:08.115177 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.115178 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.115290 (+ 112us) server_negotiation.cc:407] Connection header received
0925 13:13:08.115335 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.115335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.115347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.115391 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.115484 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51360: BlockingRecv error: recv got EOF from 127.0.0.1:51360 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:13:08.115660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9686
I20260925 13:13:08.120965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9817
I20260925 13:13:08.125494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9948
I20260925 13:13:08.129781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 9595
2026-09-25T13:13:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (609 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T13:13:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:13:08Z Disabled control of system clock
I20260925 13:13:08.156603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43091
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:13:08.236696 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:08.236958 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:08.236992 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:08.238199 10087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:13:08.238270 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:08.238291 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:08.238312 10087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:13:08.238341 10087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:13:08.239712 10087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:08.239995 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:08.240291 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:08.245182 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.245184 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.245206 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:08.245296 10087 server_base.cc:1034] running on GCE node
I20260925 13:13:08.245621 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:08.246006 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:08.247172 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790341988247160 us; error 44 us; skew 500 ppm
I20260925 13:13:08.248417 10087 webserver.cc:460] Webserver started at http://127.0.98.62:38391/ using document root <none> and password file <none>
I20260925 13:13:08.248677 10087 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:08.248744 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:08.248903 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:08.250125 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "2a596e4684464276a51c5d210102e983"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "51a7505eb41cca854b3bae920bda061e"
server_key_iv: "d4851e1aaf5a8fdb200deabbae84df67"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.250489 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a596e4684464276a51c5d210102e983"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "51a7505eb41cca854b3bae920bda061e"
server_key_iv: "d4851e1aaf5a8fdb200deabbae84df67"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.251745 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:13:08.252465 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.252627 10087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.252723 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "2a596e4684464276a51c5d210102e983"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "51a7505eb41cca854b3bae920bda061e"
server_key_iv: "d4851e1aaf5a8fdb200deabbae84df67"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.252807 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:08.256223 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:08.256484 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:08.256626 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:08.260421 10087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43091
I20260925 13:13:08.260448 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43091 every 8 connection(s)
I20260925 13:13:08.260773 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:13:08.261215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10087
I20260925 13:13:08.261296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:13:08.261508 392 external_mini_cluster.cc:1428] Setting key 7b8d7a749e36e0af611184b821f02c34
I20260925 13:13:08.261519 10154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:08.264302 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.262256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.0.98.62:43091)
0925 13:13:08.262454 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.262460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.262482 (+ 22us) server_negotiation.cc:407] Connection header received
0925 13:13:08.263020 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.263027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.263067 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.263176 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.263832 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:08.263835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:08.263836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:08.263839 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:08.263879 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:08.263890 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:08.263943 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:08.264036 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:13:08.264113 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260925 13:13:08.264859 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983: Bootstrap starting.
I20260925 13:13:08.265465 10154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:08.265806 10154 log.cc:824] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:08.266491 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983: No bootstrap required, opened a new log
I20260925 13:13:08.267797 10154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a596e4684464276a51c5d210102e983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260925 13:13:08.267935 10154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:08.267957 10154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a596e4684464276a51c5d210102e983, State: Initialized, Role: FOLLOWER
I20260925 13:13:08.268093 10154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a596e4684464276a51c5d210102e983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260925 13:13:08.268180 10154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:13:08.268224 10154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:13:08.268273 10154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:08.268795 10154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a596e4684464276a51c5d210102e983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260925 13:13:08.268908 10154 leader_election.cc:302] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a596e4684464276a51c5d210102e983; no voters:
I20260925 13:13:08.269094 10154 leader_election.cc:288] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:13:08.269366 10159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:13:08.269374 10154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:13:08.269529 10159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [term 1 LEADER]: Becoming Leader. State: Replica: 2a596e4684464276a51c5d210102e983, State: Running, Role: LEADER
I20260925 13:13:08.269626 10159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a596e4684464276a51c5d210102e983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260925 13:13:08.270047 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a596e4684464276a51c5d210102e983. Latest consensus state: current_term: 1 leader_uuid: "2a596e4684464276a51c5d210102e983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a596e4684464276a51c5d210102e983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } } }
I20260925 13:13:08.270061 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a596e4684464276a51c5d210102e983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a596e4684464276a51c5d210102e983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } } }
I20260925 13:13:08.270151 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:08.270169 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983 [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:08.270358 10168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:13:08.270862 10168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:13:08.272301 10168 catalog_manager.cc:1347] Generated new cluster ID: 96036c8ea65c4358ad25c9e73d918186
I20260925 13:13:08.272361 10168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:13:08.283942 10168 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:13:08.284119 10168 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:13:08.289723 10168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a596e4684464276a51c5d210102e983: Generated new TSK 0
I20260925 13:13:08.289983 10168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:13:08.301399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--builtin_ntp_servers=127.0.98.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:13:08.382476 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:08.382699 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:08.382720 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:08.383934 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:08.383975 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:08.384002 10178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:13:08.385367 10178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:08.385632 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:08.385970 10178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:08.390893 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.390895 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.391085 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:08.391501 10178 server_base.cc:1034] running on GCE node
I20260925 13:13:08.391724 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:08.391983 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:08.393141 10178 hybrid_clock.cc:648] HybridClock initialized: now 1790341988393115 us; error 41 us; skew 500 ppm
I20260925 13:13:08.394582 10178 webserver.cc:460] Webserver started at http://127.0.98.1:42315/ using document root <none> and password file <none>
I20260925 13:13:08.394878 10178 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:08.394946 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:08.395074 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:08.396564 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "4730c09b02164222b08e21a855d9dd73"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "6fc98ebd39fe65f7758e81ff33005167"
server_key_iv: "e24ddbdaa78411a945a7c0444c2286dc"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.396961 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4730c09b02164222b08e21a855d9dd73"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "6fc98ebd39fe65f7758e81ff33005167"
server_key_iv: "e24ddbdaa78411a945a7c0444c2286dc"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.398597 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 13:13:08.399493 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.399682 10178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:13:08.399786 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "4730c09b02164222b08e21a855d9dd73"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "6fc98ebd39fe65f7758e81ff33005167"
server_key_iv: "e24ddbdaa78411a945a7c0444c2286dc"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.399894 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:08.403398 10178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:08.403676 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:08.403828 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:08.404045 10178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:08.404367 10178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:08.404421 10178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.404455 10178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:08.404482 10178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.410560 10178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42033
I20260925 13:13:08.410609 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42033 every 8 connection(s)
I20260925 13:13:08.410930 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:13:08.414069 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.411691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42883 (local address 127.0.98.62:43091)
0925 13:13:08.411781 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.411784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.411998 (+ 214us) server_negotiation.cc:407] Connection header received
0925 13:13:08.412508 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.412508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.412513 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.412567 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.413881 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:08.413881 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:08.413882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:08.413885 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:08.413896 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:08.413900 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:08.413943 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:08.414005 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 13:13:08.414017 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 13:13:08.414337 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.411651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.98.1:42883)
0925 13:13:08.411870 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:08.411904 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:08.412411 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:08.412625 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:08.412629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:08.412637 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:08.413726 (+ 1089us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:13:08.413727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:08.413767 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:13:08.413800 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:08.413944 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:08.413947 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:08.413949 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:13:08.414012 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:13:08.414064 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260925 13:13:08.414866 10306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43091
I20260925 13:13:08.414970 10306 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:08.415122 10306 heartbeater.cc:511] Master 127.0.98.62:43091 requested a full tablet report, sending...
I20260925 13:13:08.415616 10118 ts_manager.cc:194] Registered new tserver with Master: 4730c09b02164222b08e21a855d9dd73 (127.0.98.1:42033)
I20260925 13:13:08.416410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10178
I20260925 13:13:08.416501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:13:08.416719 392 external_mini_cluster.cc:1428] Setting key 45e3a49713d44fdd5fa4abd5192a7b4d
I20260925 13:13:08.417724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--builtin_ntp_servers=127.0.98.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:13:08.498708 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:08.498934 10309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:08.498958 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:08.500108 10309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:08.500188 10309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:08.500233 10309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:13:08.501623 10309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:08.501933 10309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:08.502278 10309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:08.507325 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.507337 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.507351 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:08.507570 10309 server_base.cc:1034] running on GCE node
I20260925 13:13:08.507839 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:08.508076 10309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:08.509246 10309 hybrid_clock.cc:648] HybridClock initialized: now 1790341988509247 us; error 41 us; skew 500 ppm
I20260925 13:13:08.510581 10309 webserver.cc:460] Webserver started at http://127.0.98.2:34485/ using document root <none> and password file <none>
I20260925 13:13:08.510823 10309 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:08.510892 10309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:08.511025 10309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:08.512174 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "725e392101604274b71d540fd4ad1b19"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "add37d25a85c961efe290ec6101bc355"
server_key_iv: "14d1a37a1f760f6c1fde2191ea36f3a2"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.512540 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "725e392101604274b71d540fd4ad1b19"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "add37d25a85c961efe290ec6101bc355"
server_key_iv: "14d1a37a1f760f6c1fde2191ea36f3a2"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.513918 10309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:13:08.514660 10323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.514873 10309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:13:08.514969 10309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "725e392101604274b71d540fd4ad1b19"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "add37d25a85c961efe290ec6101bc355"
server_key_iv: "14d1a37a1f760f6c1fde2191ea36f3a2"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.515048 10309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:08.518378 10309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:08.518628 10309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:08.518769 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:08.518985 10309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:08.519276 10309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:08.519326 10309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.519366 10309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:08.519393 10309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.524979 10309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34821
I20260925 13:13:08.525028 10436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34821 every 8 connection(s)
I20260925 13:13:08.525369 10309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:13:08.528468 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.526152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55479 (local address 127.0.98.62:43091)
0925 13:13:08.526218 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.526220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.526461 (+ 241us) server_negotiation.cc:407] Connection header received
0925 13:13:08.526964 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.526966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.526972 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.527022 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.528175 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:08.528176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:08.528177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:08.528181 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:08.528196 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:08.528203 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:08.528271 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:08.528373 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 13:13:08.528389 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:13:08.528671 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.526136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.98.2:55479)
0925 13:13:08.526331 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:08.526364 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:08.526853 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:08.527090 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:08.527096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:08.527109 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:08.528010 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:13:08.528011 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:08.528038 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:13:08.528067 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:08.528268 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:08.528274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:08.528277 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:13:08.528372 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 13:13:08.528435 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260925 13:13:08.529151 10437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43091
I20260925 13:13:08.529225 10437 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:08.529389 10437 heartbeater.cc:511] Master 127.0.98.62:43091 requested a full tablet report, sending...
I20260925 13:13:08.529781 10118 ts_manager.cc:194] Registered new tserver with Master: 725e392101604274b71d540fd4ad1b19 (127.0.98.2:34821)
I20260925 13:13:08.532310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10309
I20260925 13:13:08.532387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:13:08.532591 392 external_mini_cluster.cc:1428] Setting key 87f9570f8276bc34d40324ec3a31e97f
I20260925 13:13:08.533457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--builtin_ntp_servers=127.0.98.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:13:08.615831 10440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:08.616046 10440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:08.616070 10440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:08.617268 10440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:08.617307 10440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:08.617333 10440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:13:08.618888 10440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:08.619143 10440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:08.619447 10440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:08.624369 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.624388 10445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:08.624360 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:08.624686 10440 server_base.cc:1034] running on GCE node
I20260925 13:13:08.624894 10440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:08.625145 10440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:08.626307 10440 hybrid_clock.cc:648] HybridClock initialized: now 1790341988626296 us; error 28 us; skew 500 ppm
I20260925 13:13:08.627513 10440 webserver.cc:460] Webserver started at http://127.0.98.3:39929/ using document root <none> and password file <none>
I20260925 13:13:08.627748 10440 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:08.627794 10440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:08.627945 10440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:08.628997 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "a2933967f4c1405fb130ac4f11433491"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "9ed4f9bb04c87c449333e97a365a943b"
server_key_iv: "6d2de65db37161f825afa64dd532e608"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.629352 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a2933967f4c1405fb130ac4f11433491"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "9ed4f9bb04c87c449333e97a365a943b"
server_key_iv: "6d2de65db37161f825afa64dd532e608"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.630697 10440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:13:08.631408 10454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.631589 10440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:13:08.631659 10440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "a2933967f4c1405fb130ac4f11433491"
format_stamp: "Formatted at 2026-09-25 13:13:08 on dist-test-slave-2d0s"
server_key: "9ed4f9bb04c87c449333e97a365a943b"
server_key_iv: "6d2de65db37161f825afa64dd532e608"
server_key_version: "encryptionkey@0"
I20260925 13:13:08.631750 10440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:08.635035 10440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:08.635281 10440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:08.635408 10440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:08.635612 10440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:08.635890 10440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:08.635919 10440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.635967 10440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:08.635994 10440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:08.641631 10440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40485
I20260925 13:13:08.641716 10567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40485 every 8 connection(s)
I20260925 13:13:08.642016 10440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:13:08.645205 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.642982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58545 (local address 127.0.98.62:43091)
0925 13:13:08.643083 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.643086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.643465 (+ 379us) server_negotiation.cc:407] Connection header received
0925 13:13:08.643886 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.643887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.643892 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.643951 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.645005 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:08.645006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:08.645006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:13:08.645010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:08.645022 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:13:08.645026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:08.645072 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:08.645142 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 13:13:08.645153 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 13:13:08.645367 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:08.642756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.98.3:58545)
0925 13:13:08.643319 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:08.643352 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:08.643802 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:08.644002 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:08.644005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:08.644012 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:08.644871 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:13:08.644871 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:08.644899 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:13:08.644926 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:08.645071 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:08.645073 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:08.645075 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:13:08.645126 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:13:08.645170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":60,"threads_started":1}
I20260925 13:13:08.645946 10568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43091
I20260925 13:13:08.646019 10568 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:08.646181 10568 heartbeater.cc:511] Master 127.0.98.62:43091 requested a full tablet report, sending...
I20260925 13:13:08.646552 10118 ts_manager.cc:194] Registered new tserver with Master: a2933967f4c1405fb130ac4f11433491 (127.0.98.3:40485)
I20260925 13:13:08.648304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10440
I20260925 13:13:08.648387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:13:08.648608 392 external_mini_cluster.cc:1428] Setting key b4fed3912ee2566eb919c3501c70be11
I20260925 13:13:08.649390 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 13:13:08.651198 10157 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:13:08.650769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.98.62:43091)
0925 13:13:08.650844 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:08.650846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:08.650941 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:13:08.650977 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:08.650978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:08.650982 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:08.651026 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:08.651141 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37754: BlockingRecv error: recv got EOF from 127.0.0.1:37754 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 13:13:08.651257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10178
I20260925 13:13:08.655793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10309
I20260925 13:13:08.660024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10440
I20260925 13:13:08.664238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10087
2026-09-25T13:13:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (534 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:13:08 dist-test-slave-2d0s krb5kdc[10578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:13:08 dist-test-slave-2d0s krb5kdc[10578](info): set up 2 sockets
Sep 25 13:13:08 dist-test-slave-2d0s krb5kdc[10578](info): commencing operation
krb5kdc: starting...
W20260925 13:13:09.690510 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:13:09 dist-test-slave-2d0s krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:13:09.809170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:09.887562 10591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:09.887784 10591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:09.887818 10591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:09.888934 10591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:13:09.888995 10591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:09.889020 10591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:09.889050 10591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:13:09.889078 10591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:13:09.890583 10591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:09.890839 10591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:09.891060 10591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:09.895969 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:09.895965 10599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:09.896023 10591 server_base.cc:1034] running on GCE node
W20260925 13:13:09.896025 10597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:09.896386 10591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:09.896663 10591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:09.897807 10591 hybrid_clock.cc:648] HybridClock initialized: now 1790341989897780 us; error 49 us; skew 500 ppm
Sep 25 13:13:09 dist-test-slave-2d0s krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:09.900214 10591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:13:09.900564 10591 webserver.cc:460] Webserver started at http://127.0.98.62:40641/ using document root <none> and password file <none>
I20260925 13:13:09.900748 10591 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:09.900786 10591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:09.900876 10591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:09.901944 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "5c5cf373b84241889c47c0ba863d91ef"
format_stamp: "Formatted at 2026-09-25 13:13:09 on dist-test-slave-2d0s"
server_key: "633c340c717b3b32ca37c8b4ddbc3cf7"
server_key_iv: "c67cbaba1af7ad0a469e64a804dcd84a"
server_key_version: "encryptionkey@0"
I20260925 13:13:09.902315 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c5cf373b84241889c47c0ba863d91ef"
format_stamp: "Formatted at 2026-09-25 13:13:09 on dist-test-slave-2d0s"
server_key: "633c340c717b3b32ca37c8b4ddbc3cf7"
server_key_iv: "c67cbaba1af7ad0a469e64a804dcd84a"
server_key_version: "encryptionkey@0"
I20260925 13:13:09.903590 10591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:13:09.904286 10606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:09.904451 10591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:13:09.904543 10591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "5c5cf373b84241889c47c0ba863d91ef"
format_stamp: "Formatted at 2026-09-25 13:13:09 on dist-test-slave-2d0s"
server_key: "633c340c717b3b32ca37c8b4ddbc3cf7"
server_key_iv: "c67cbaba1af7ad0a469e64a804dcd84a"
server_key_version: "encryptionkey@0"
I20260925 13:13:09.904614 10591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:09.922720 10591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:09.924978 10591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:09.925130 10591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:09.928906 10591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35283
I20260925 13:13:09.928943 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35283 every 8 connection(s)
I20260925 13:13:09.929287 10591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:13:09.930269 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:09.932425 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: Bootstrap starting.
I20260925 13:13:09.932935 10659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:09.933199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10591
I20260925 13:13:09.933274 10659 log.cc:824] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:09.933308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:13:09.933525 392 external_mini_cluster.cc:1428] Setting key 49161e265b511118e01de29ef79616dd
I20260925 13:13:09.934167 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: No bootstrap required, opened a new log
I20260925 13:13:09.936647 10659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cf373b84241889c47c0ba863d91ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260925 13:13:09.936815 10659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:09.936851 10659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5cf373b84241889c47c0ba863d91ef, State: Initialized, Role: FOLLOWER
I20260925 13:13:09.936970 10659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cf373b84241889c47c0ba863d91ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260925 13:13:09.937054 10659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:13:09.937083 10659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:13:09.937125 10659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:09.937790 10659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cf373b84241889c47c0ba863d91ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260925 13:13:09.937906 10659 leader_election.cc:302] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c5cf373b84241889c47c0ba863d91ef; no voters:
I20260925 13:13:09.938057 10659 leader_election.cc:288] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:13:09.938304 10659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:13:09.938356 10664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:13:09.938464 10664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [term 1 LEADER]: Becoming Leader. State: Replica: 5c5cf373b84241889c47c0ba863d91ef, State: Running, Role: LEADER
I20260925 13:13:09.938556 10664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cf373b84241889c47c0ba863d91ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260925 13:13:09.939010 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c5cf373b84241889c47c0ba863d91ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cf373b84241889c47c0ba863d91ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260925 13:13:09.939124 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:09.939003 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c5cf373b84241889c47c0ba863d91ef. Latest consensus state: current_term: 1 leader_uuid: "5c5cf373b84241889c47c0ba863d91ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cf373b84241889c47c0ba863d91ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260925 13:13:09.939291 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef [sys.catalog]: This master's current role is: LEADER
Sep 25 13:13:09 dist-test-slave-2d0s krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260925 13:13:09.940090 10679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:13:09.940677 10679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:13:09.940752 10673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:13:09.940914 10673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:13:09.942622 10673 catalog_manager.cc:1347] Generated new cluster ID: 7a177e4a55f047a5ad4dd2b49b910600
I20260925 13:13:09.942669 10673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:13:09.944679 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:09.934092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.0.98.62:35283)
0925 13:13:09.934253 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:09.934261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:09.934293 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:13:09.934690 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:09.934696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:09.934740 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:09.934845 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:09.935599 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:09.936434 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:09.937490 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:09.937998 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:09.938108 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:09.940909 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:09.940913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:09.940913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:09.940923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:09.942903 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:09.943262 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:09.943265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:09.943266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:09.943353 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:09.943538 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:09.943538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:09.943538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:09.943783 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:09.944286 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:09.944355 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 13:13:09.944477 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 13:13:09.949095 10673 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:13:09.949644 10673 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:13:09.960305 10673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: Generated new TSK 0
I20260925 13:13:09.960484 10673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:13:09.988766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:10.072222 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:10.072453 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:10.072489 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:10.073609 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:10.073674 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:10.073751 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:13:10.075163 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:10.075440 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:10.075676 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:10.080634 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:10.080636 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:10.080683 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:10.081002 10685 server_base.cc:1034] running on GCE node
I20260925 13:13:10.081207 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:10.081441 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:10.082592 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790341990082572 us; error 31 us; skew 500 ppm
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:10.085059 10685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:13:10.085453 10685 webserver.cc:460] Webserver started at http://127.0.98.1:44265/ using document root <none> and password file <none>
I20260925 13:13:10.085726 10685 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:10.085793 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:10.085938 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:10.086951 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a1a4546261044cd9e444f833d56bd85"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "c2e3fe47becb417f8bc0598f874d9bd4"
server_key_iv: "e63f537643ef845ab3173494fc5d3588"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.087301 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a1a4546261044cd9e444f833d56bd85"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "c2e3fe47becb417f8bc0598f874d9bd4"
server_key_iv: "e63f537643ef845ab3173494fc5d3588"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.088578 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:13:10.089268 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.089442 10685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:13:10.089535 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "3a1a4546261044cd9e444f833d56bd85"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "c2e3fe47becb417f8bc0598f874d9bd4"
server_key_iv: "e63f537643ef845ab3173494fc5d3588"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.089622 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:10.112149 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:10.114327 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:10.114493 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:10.114722 10685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:10.115064 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:10.115119 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.115161 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:10.115198 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.120797 10685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42307
I20260925 13:13:10.120913 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42307 every 8 connection(s)
I20260925 13:13:10.121209 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:13:10.123601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10685
I20260925 13:13:10.123730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:13:10.123915 392 external_mini_cluster.cc:1428] Setting key e8c9d46d94e16b55a1ea73a5ad67b1fe
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:13:10.131250 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.122002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34451 (local address 127.0.98.62:35283)
0925 13:13:10.122051 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.122072 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.122263 (+ 191us) server_negotiation.cc:407] Connection header received
0925 13:13:10.122808 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.122810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.122822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.122869 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:10.123920 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.124641 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.125704 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.125933 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.125991 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.128375 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:10.128377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:10.128377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:10.128396 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:10.129861 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.130252 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.130253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.130254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.130310 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.130573 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.130574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.130574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.130709 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:10.130783 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.131063 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.131157 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:13:10.131436 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.121958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:34451)
0925 13:13:10.122120 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.122171 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.122697 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.122963 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.122966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.123552 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:10.123799 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.123806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.124731 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.124732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.125596 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.125598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.126017 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.126018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.126168 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.126698 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:10.126708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:10.128269 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:10.129950 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:10.129957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:10.129959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:10.130168 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:10.130371 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:10.130372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:10.130373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:10.130500 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:10.130775 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:10.130781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:10.130931 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.131043 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.131195 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:13:10.132050 10814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260925 13:13:10.132138 10814 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:10.132278 10814 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260925 13:13:10.132875 10623 ts_manager.cc:194] Registered new tserver with Master: 3a1a4546261044cd9e444f833d56bd85 (127.0.98.1:42307)
I20260925 13:13:10.133468 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:13:10.148728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:10.228026 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:10.228245 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:10.228268 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:10.229396 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:10.229437 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:10.229463 10819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:13:10.230926 10819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:10.231149 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:10.231391 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:10.236289 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:10.236296 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:10.236325 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:10.236498 10819 server_base.cc:1034] running on GCE node
I20260925 13:13:10.236709 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:10.236948 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:10.238116 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790341990238109 us; error 59 us; skew 500 ppm
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:10.240831 10819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:13:10.241330 10819 webserver.cc:460] Webserver started at http://127.0.98.2:38245/ using document root <none> and password file <none>
I20260925 13:13:10.241554 10819 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:10.241623 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:10.241801 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:10.242851 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "062efb386dc74be4a9129e6331805061"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "6b508e0588b73d7bd9777291448f073c"
server_key_iv: "4468439b5332b0e7b545f2a9a7fb7073"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.243232 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "062efb386dc74be4a9129e6331805061"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "6b508e0588b73d7bd9777291448f073c"
server_key_iv: "4468439b5332b0e7b545f2a9a7fb7073"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.244532 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:13:10.245296 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.245494 10819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:13:10.245591 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "062efb386dc74be4a9129e6331805061"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "6b508e0588b73d7bd9777291448f073c"
server_key_iv: "4468439b5332b0e7b545f2a9a7fb7073"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.245695 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:10.261018 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:10.263036 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:10.263212 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:10.263425 10819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:10.263736 10819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:10.263789 10819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.263828 10819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:10.263864 10819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.269398 10819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37081
I20260925 13:13:10.269447 10947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37081 every 8 connection(s)
I20260925 13:13:10.269743 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:13:10.273320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10819
I20260925 13:13:10.273408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:13:10.273597 392 external_mini_cluster.cc:1428] Setting key 417aa42fa29d1751f35d58bb6ea52d16
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:13:10.279505 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.270374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55607 (local address 127.0.98.62:35283)
0925 13:13:10.270440 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.270442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.270703 (+ 261us) server_negotiation.cc:407] Connection header received
0925 13:13:10.271226 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.271227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.271239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.271285 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:10.272227 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.272729 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.273950 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.274195 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.274267 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.276483 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:10.276485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:10.276485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:10.276492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:10.278021 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.278509 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.278509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.278510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.278556 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.278885 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.278885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.278885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.278981 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:10.279030 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.279380 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.279441 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:13:10.279830 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.270361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:55607)
0925 13:13:10.270521 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.270586 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.271119 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.271348 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.271352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.271941 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:10.272125 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.272129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.272823 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.272824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.273854 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.273857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.274288 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.274290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.274428 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.274970 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:10.274978 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:10.276400 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:10.278101 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:10.278108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:10.278110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:10.278431 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:10.278619 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:10.278620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:10.278620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:10.278781 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:10.279054 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:10.279061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:10.279234 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.279355 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.279544 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260925 13:13:10.280333 10948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260925 13:13:10.280472 10948 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:10.280644 10948 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260925 13:13:10.281032 10623 ts_manager.cc:194] Registered new tserver with Master: 062efb386dc74be4a9129e6331805061 (127.0.98.2:37081)
I20260925 13:13:10.281397 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55607
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:13:10.300837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:10.380852 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:10.381042 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:10.381064 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:10.382129 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:10.382162 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:10.382186 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:13:10.383608 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:10.383838 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:10.384012 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:10.388778 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:10.388813 10953 server_base.cc:1034] running on GCE node
W20260925 13:13:10.388816 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:10.388779 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:10.389086 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:10.389266 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:10.390398 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790341990390386 us; error 26 us; skew 500 ppm
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:10.392962 10953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:13:10.393378 10953 webserver.cc:460] Webserver started at http://127.0.98.3:33741/ using document root <none> and password file <none>
I20260925 13:13:10.393543 10953 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:10.393581 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:10.393726 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:13:10.394657 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "278062db030342d2a372325908722083"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "84ddf30ea912564eddd89fdc7bb949b5"
server_key_iv: "56eb1a34f8c9a1f477e4421095b3f571"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.394992 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "278062db030342d2a372325908722083"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "84ddf30ea912564eddd89fdc7bb949b5"
server_key_iv: "56eb1a34f8c9a1f477e4421095b3f571"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.396225 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:13:10.396858 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.397032 10953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:13:10.397127 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "278062db030342d2a372325908722083"
format_stamp: "Formatted at 2026-09-25 13:13:10 on dist-test-slave-2d0s"
server_key: "84ddf30ea912564eddd89fdc7bb949b5"
server_key_iv: "56eb1a34f8c9a1f477e4421095b3f571"
server_key_version: "encryptionkey@0"
I20260925 13:13:10.397204 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:10.412215 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:10.414491 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:10.414644 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:10.414865 10953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:10.415197 10953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:10.415248 10953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.415288 10953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:10.415314 10953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:10.421105 10953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37651
I20260925 13:13:10.421146 11081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37651 every 8 connection(s)
I20260925 13:13:10.421450 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:13:10.425300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 10953
I20260925 13:13:10.425457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:13:10.425653 392 external_mini_cluster.cc:1428] Setting key aef7d92483387c64f7f2b5f65193639f
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:13:10.430562 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.422176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39917 (local address 127.0.98.62:35283)
0925 13:13:10.422239 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.422241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.422436 (+ 195us) server_negotiation.cc:407] Connection header received
0925 13:13:10.422899 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.422901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.422910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.422956 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:10.423836 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.424325 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.425307 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.425616 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.425720 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.427941 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:10.427943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:10.427943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:10.427950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:10.429137 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.429556 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.429557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.429558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.429614 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.429907 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.429908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.429909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.430024 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:10.430098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.430355 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.430447 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:13:10.430784 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.422174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:39917)
0925 13:13:10.422314 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.422347 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.422796 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.423019 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.423022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.423577 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:10.423763 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.423768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.424411 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.424412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.425203 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.425205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.425730 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.425731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.425875 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.426419 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:10.426428 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:10.427832 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:10.429246 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:10.429252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:10.429254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:10.429479 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:10.429703 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:10.429704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:10.429704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:10.429812 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:10.430090 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:10.430095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:10.430230 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.430336 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.430559 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260925 13:13:10.431226 11082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260925 13:13:10.431298 11082 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:10.431489 11082 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260925 13:13:10.431854 10623 ts_manager.cc:194] Registered new tserver with Master: 278062db030342d2a372325908722083 (127.0.98.3:37651)
I20260925 13:13:10.432220 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39917
I20260925 13:13:10.437354 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:13:10.444962 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.438906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.0.98.62:35283)
0925 13:13:10.438982 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.438984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.439079 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:13:10.439201 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.439202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.439213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.439283 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:10.439997 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.440504 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.441224 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.441404 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.441453 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.442620 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:10.442621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:10.442621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:10.442627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:10.444010 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.444324 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.444325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.444326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.444390 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:10.444562 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:10.444562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:10.444563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:10.444699 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:10.444770 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.444847 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.444902 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 13:13:10.446504 10623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:13:10.448051 10623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:13:10.453969 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.450489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.98.3:37651)
0925 13:13:10.450649 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.450654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.450664 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:13:10.450698 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.450700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.450717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.450812 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:10.451015 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.451820 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.452968 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.453550 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.453637 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.453744 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.453852 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.453889 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260925 13:13:10.453960 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.450178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37651 (local address 127.0.0.1:50032)
0925 13:13:10.450424 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.450433 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.450473 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.450842 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.450845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.450847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:10.450929 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.450931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.451950 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.451951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.452853 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.452855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.453635 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.453636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.453751 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.453767 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.453839 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.453894 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":24,"thread_start_us":57,"threads_started":1}
I20260925 13:13:10.454439 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.450112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37081 (local address 127.0.0.1:38096)
0925 13:13:10.450304 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.450343 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.450406 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.450861 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.450862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.450864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:10.450945 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.450947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.451975 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.451976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.453345 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.453347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.454232 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.454233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.454297 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.454301 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.454356 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.454396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":89,"thread_start_us":89,"threads_started":1}
I20260925 13:13:10.454685 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.450242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.98.2:37081)
0925 13:13:10.450689 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.450693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.450704 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:13:10.450744 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.450747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.450762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.451010 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:10.451078 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.451868 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.453427 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.454150 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.454219 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.454267 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.454558 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.454602 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":89,"threads_started":1}
I20260925 13:13:10.455044 11016 tablet_service.cc:1467] Processing CreateTablet for tablet da8ecbe2ddc943adb754eec1a41580da (DEFAULT_TABLE table=test-table [id=73da2d5899fb44eab5080d6549441339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:13:10.455245 10882 tablet_service.cc:1467] Processing CreateTablet for tablet da8ecbe2ddc943adb754eec1a41580da (DEFAULT_TABLE table=test-table [id=73da2d5899fb44eab5080d6549441339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:13:10.455421 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8ecbe2ddc943adb754eec1a41580da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:10.455447 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.450361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42307 (local address 127.0.0.1:50442)
0925 13:13:10.450648 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.450659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.450724 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.451950 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.451951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.451953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:10.451990 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.451992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.453219 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.453220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.454303 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.454305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.455206 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.455207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.455265 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.455269 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.455316 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.455375 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260925 13:13:10.455570 10882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8ecbe2ddc943adb754eec1a41580da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:10.455706 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.450485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.0.98.1:42307)
0925 13:13:10.450649 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.450654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.450709 (+ 55us) server_negotiation.cc:407] Connection header received
0925 13:13:10.450899 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.450901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.450918 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.450996 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:10.452054 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.453120 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.454404 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.455055 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.455122 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.455172 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.455593 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.455633 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260925 13:13:10.456270 10748 tablet_service.cc:1467] Processing CreateTablet for tablet da8ecbe2ddc943adb754eec1a41580da (DEFAULT_TABLE table=test-table [id=73da2d5899fb44eab5080d6549441339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:13:10.456590 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8ecbe2ddc943adb754eec1a41580da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:10.458259 11096 tablet_bootstrap.cc:492] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Bootstrap starting.
I20260925 13:13:10.458756 11097 tablet_bootstrap.cc:492] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Bootstrap starting.
I20260925 13:13:10.459075 11096 tablet_bootstrap.cc:654] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:10.459450 11097 tablet_bootstrap.cc:654] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:10.459687 11096 log.cc:824] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:10.459685 11098 tablet_bootstrap.cc:492] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: Bootstrap starting.
I20260925 13:13:10.460002 11097 log.cc:824] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:10.460400 11098 tablet_bootstrap.cc:654] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:10.460461 11096 tablet_bootstrap.cc:492] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: No bootstrap required, opened a new log
I20260925 13:13:10.460536 11096 ts_tablet_manager.cc:1397] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:13:10.460865 11097 tablet_bootstrap.cc:492] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: No bootstrap required, opened a new log
I20260925 13:13:10.460942 11097 ts_tablet_manager.cc:1397] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:13:10.460902 11098 log.cc:824] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:10.461616 11098 tablet_bootstrap.cc:492] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: No bootstrap required, opened a new log
I20260925 13:13:10.461720 11098 ts_tablet_manager.cc:1397] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:13:10.463290 11097 raft_consensus.cc:348] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.463464 11097 raft_consensus.cc:374] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:10.463505 11097 raft_consensus.cc:729] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278062db030342d2a372325908722083, State: Initialized, Role: FOLLOWER
I20260925 13:13:10.463419 11098 raft_consensus.cc:348] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.463586 11098 raft_consensus.cc:374] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:10.463621 11097 consensus_queue.cc:260] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.463667 11098 raft_consensus.cc:729] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1a4546261044cd9e444f833d56bd85, State: Initialized, Role: FOLLOWER
I20260925 13:13:10.463773 11098 consensus_queue.cc:260] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.463758 11096 raft_consensus.cc:348] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.463867 11097 ts_tablet_manager.cc:1428] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:13:10.463910 11096 raft_consensus.cc:374] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:10.463949 11096 raft_consensus.cc:729] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062efb386dc74be4a9129e6331805061, State: Initialized, Role: FOLLOWER
I20260925 13:13:10.463996 11082 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
I20260925 13:13:10.464064 11098 ts_tablet_manager.cc:1428] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:13:10.464063 11096 consensus_queue.cc:260] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.464156 10814 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
I20260925 13:13:10.464309 11096 ts_tablet_manager.cc:1428] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 13:13:10.464321 10948 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
W20260925 13:13:10.520340 10949 tablet.cc:2369] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:13:10.621934 10815 tablet.cc:2369] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:13:10.671984 11083 tablet.cc:2369] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:13:10.781852 11103 raft_consensus.cc:482] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:13:10.781988 11103 raft_consensus.cc:504] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.782292 11103 leader_election.cc:288] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062efb386dc74be4a9129e6331805061 (127.0.98.2:37081), 278062db030342d2a372325908722083 (127.0.98.3:37651)
I20260925 13:13:10.785329 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.782576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38471 (local address 127.0.98.2:37081)
0925 13:13:10.782623 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.782625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.782712 (+ 87us) server_negotiation.cc:407] Connection header received
0925 13:13:10.782765 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.782766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.782778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.782835 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:10.783010 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.783543 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.784644 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.784968 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.785051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.785060 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.785224 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.785257 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:13:10.785333 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.782587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35669 (local address 127.0.98.3:37651)
0925 13:13:10.782648 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.782650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.782786 (+ 136us) server_negotiation.cc:407] Connection header received
0925 13:13:10.782858 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.782859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.782871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.782929 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:10.783148 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.783633 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.784590 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.784968 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.785051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.785061 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.785226 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.785263 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:13:10.785363 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.782501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37651 (local address 127.0.98.1:35669)
0925 13:13:10.782673 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.782701 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.782780 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.782960 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.782961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.782963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:10.783053 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.783055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.783727 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.783727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.784488 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.784490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.785057 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.785059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.785130 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.785133 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.785219 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.785257 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260925 13:13:10.785365 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.782501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37081 (local address 127.0.98.1:38471)
0925 13:13:10.782626 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.782639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:10.782714 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:10.782854 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:10.782855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:10.782858 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:10.782939 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.782941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.783619 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.783620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.784500 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:10.784502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.785055 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.785056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:10.785129 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.785133 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:13:10.785218 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 13:13:10.785262 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":19,"thread_start_us":61,"threads_started":1}
I20260925 13:13:10.785811 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8ecbe2ddc943adb754eec1a41580da" candidate_uuid: "3a1a4546261044cd9e444f833d56bd85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278062db030342d2a372325908722083" is_pre_election: true
I20260925 13:13:10.786032 11036 raft_consensus.cc:2454] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a1a4546261044cd9e444f833d56bd85 in term 0.
I20260925 13:13:10.786085 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8ecbe2ddc943adb754eec1a41580da" candidate_uuid: "3a1a4546261044cd9e444f833d56bd85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062efb386dc74be4a9129e6331805061" is_pre_election: true
I20260925 13:13:10.786283 10902 raft_consensus.cc:2454] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a1a4546261044cd9e444f833d56bd85 in term 0.
I20260925 13:13:10.786280 10701 leader_election.cc:302] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278062db030342d2a372325908722083, 3a1a4546261044cd9e444f833d56bd85; no voters:
I20260925 13:13:10.786412 11103 raft_consensus.cc:2781] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:13:10.786476 11103 raft_consensus.cc:482] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:13:10.786504 11103 raft_consensus.cc:3037] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:10.787214 11103 raft_consensus.cc:504] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.787362 11103 leader_election.cc:288] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [CANDIDATE]: Term 1 election: Requested vote from peers 062efb386dc74be4a9129e6331805061 (127.0.98.2:37081), 278062db030342d2a372325908722083 (127.0.98.3:37651)
I20260925 13:13:10.787529 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8ecbe2ddc943adb754eec1a41580da" candidate_uuid: "3a1a4546261044cd9e444f833d56bd85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062efb386dc74be4a9129e6331805061"
I20260925 13:13:10.787583 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8ecbe2ddc943adb754eec1a41580da" candidate_uuid: "3a1a4546261044cd9e444f833d56bd85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278062db030342d2a372325908722083"
I20260925 13:13:10.787627 10902 raft_consensus.cc:3037] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:10.787686 11036 raft_consensus.cc:3037] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:10.788328 10902 raft_consensus.cc:2454] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a1a4546261044cd9e444f833d56bd85 in term 1.
I20260925 13:13:10.788354 11036 raft_consensus.cc:2454] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a1a4546261044cd9e444f833d56bd85 in term 1.
I20260925 13:13:10.788508 10701 leader_election.cc:302] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278062db030342d2a372325908722083, 3a1a4546261044cd9e444f833d56bd85; no voters:
I20260925 13:13:10.788622 11103 raft_consensus.cc:2781] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:13:10.788786 11103 raft_consensus.cc:686] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 1 LEADER]: Becoming Leader. State: Replica: 3a1a4546261044cd9e444f833d56bd85, State: Running, Role: LEADER
I20260925 13:13:10.788895 11103 consensus_queue.cc:237] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } }
I20260925 13:13:10.789719 10621 catalog_manager.cc:5495] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a1a4546261044cd9e444f833d56bd85 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3a1a4546261044cd9e444f833d56bd85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a1a4546261044cd9e444f833d56bd85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260925 13:13:10.809187 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.806865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.0.98.1:42307)
0925 13:13:10.806930 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:10.806932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:10.806941 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:13:10.806983 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:10.806984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:10.807009 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:10.807080 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:13:10.807183 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.807707 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.808434 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:10.808650 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:10.808725 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:10.808889 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:13:10.808982 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:13:10.809074 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:10.809093 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 13:13:10.809110 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:13:10.810940 11036 raft_consensus.cc:1261] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 1 FOLLOWER]: Refusing update from remote peer 3a1a4546261044cd9e444f833d56bd85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:13:10.810940 10902 raft_consensus.cc:1261] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 1 FOLLOWER]: Refusing update from remote peer 3a1a4546261044cd9e444f833d56bd85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:13:10.811311 11107 consensus_queue.cc:1035] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "062efb386dc74be4a9129e6331805061" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:13:10.811463 11103 consensus_queue.cc:1035] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "278062db030342d2a372325908722083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:13:10.814107 11112 mvcc.cc:205] Tried to move back new op lower bound from 7333240794358820864 to 7333240794271887360. Current Snapshot: MvccSnapshot[applied={T|T < 7333240794358820864}]
I20260925 13:13:10.815902 10621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:13:10.816013 10621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:13:10.816794 10621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c5cf373b84241889c47c0ba863d91ef: Sending DeleteTablet for 3 replicas of tablet da8ecbe2ddc943adb754eec1a41580da
I20260925 13:13:10.817119 10748 tablet_service.cc:1514] Processing DeleteTablet for tablet da8ecbe2ddc943adb754eec1a41580da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:13:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50442
I20260925 13:13:10.817162 10882 tablet_service.cc:1514] Processing DeleteTablet for tablet da8ecbe2ddc943adb754eec1a41580da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:13:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38096
I20260925 13:13:10.817195 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet da8ecbe2ddc943adb754eec1a41580da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:13:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50032
I20260925 13:13:10.817364 11119 tablet_replica.cc:330] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85: stopping tablet replica
I20260925 13:13:10.817466 11119 raft_consensus.cc:2229] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:13:10.817574 11119 raft_consensus.cc:2258] T da8ecbe2ddc943adb754eec1a41580da P 3a1a4546261044cd9e444f833d56bd85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:13:10.817571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10685
I20260925 13:13:10.821791 11121 tablet_replica.cc:330] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: stopping tablet replica
I20260925 13:13:10.821796 11120 tablet_replica.cc:330] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: stopping tablet replica
I20260925 13:13:10.821902 11120 raft_consensus.cc:2229] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:13:10.821935 11121 raft_consensus.cc:2229] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:13:10.822024 11120 raft_consensus.cc:2258] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:13:10.822048 11121 raft_consensus.cc:2258] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:13:10.822494 11120 ts_tablet_manager.cc:1905] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:13:10.822485 11121 ts_tablet_manager.cc:1905] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 13:13:10.823607 10609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42307 (error 108)
I20260925 13:13:10.824036 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.823916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42307 (local address 127.0.0.1:50468)
0925 13:13:10.823969 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.824001 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 13:13:10.824134 10609 catalog_manager.cc:4584] TS 3a1a4546261044cd9e444f833d56bd85 (127.0.98.1:42307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da8ecbe2ddc943adb754eec1a41580da: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42307: connect: Connection refused (error 111)
I20260925 13:13:10.824312 11121 ts_tablet_manager.cc:1918] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:13:10.824312 11120 ts_tablet_manager.cc:1918] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:13:10.824389 11121 log.cc:1196] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/da8ecbe2ddc943adb754eec1a41580da
I20260925 13:13:10.824389 11120 log.cc:1196] T da8ecbe2ddc943adb754eec1a41580da P 062efb386dc74be4a9129e6331805061: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/da8ecbe2ddc943adb754eec1a41580da
I20260925 13:13:10.824456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10819
I20260925 13:13:10.825172 11121 ts_tablet_manager.cc:1939] T da8ecbe2ddc943adb754eec1a41580da P 278062db030342d2a372325908722083: Deleting consensus metadata
I20260925 13:13:10.825524 10607 catalog_manager.cc:4841] TS 278062db030342d2a372325908722083 (127.0.98.3:37651): tablet da8ecbe2ddc943adb754eec1a41580da (table test-table [id=73da2d5899fb44eab5080d6549441339]) successfully deleted
I20260925 13:13:10.829423 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:10.829318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37081 (local address 127.0.0.1:38104)
0925 13:13:10.829366 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:10.829396 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:13:10.829450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10953
I20260925 13:13:10.834295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 10591
2026-09-25T13:13:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2170 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[11125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[11125](info): set up 2 sockets
Sep 25 13:13:10 dist-test-slave-2d0s krb5kdc[11125](info): commencing operation
krb5kdc: starting...
W20260925 13:13:11.906620 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:13:11 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790341991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:13:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 13:13:12.070354 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpOnpMs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/postgres -l logfile start
2026-09-25 13:13:14.907 UTC [11155] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 13:13:14.908 UTC [11155] LOG: listening on IPv4 address "127.0.98.20", port 54173
2026-09-25 13:13:14.911 UTC [11155] LOG: listening on Unix socket "/tmp/.s.PGSQL.54173"
2026-09-25 13:13:14.946 UTC [11158] LOG: database system was shut down at 2026-09-25 13:13:13 UTC
2026-09-25 13:13:14.950 UTC [11155] LOG: database system is ready to accept connections
I20260925 13:13:15.882527 392 mini_postgres.cc:96] Postgres bound to 54173
2026-09-25 13:13:15.888 UTC [11166] FATAL: database "slave" does not exist
127.0.98.20:54173 - accepting connections
I20260925 13:13:15.889173 392 mini_ranger.cc:161] Starting Ranger...
I20260925 13:13:15.903259 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 13:13:16.370635 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 13:13:16.370738 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-admin
2026-09-25 13:13:16,646 [I] DB FLAVOR :POSTGRES
2026-09-25 13:13:16,646 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:13:16,647 [I] Checking connection..
2026-09-25 13:13:16,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 13:13:16,910 [I] Checking connection passed.
2026-09-25 13:13:16,910 [I] --------- Verifying version history table ---------
2026-09-25 13:13:16,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:13:17,141 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:13:17,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:13:17,396 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:13:17,396 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 13:13:17,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 13:13:17.620 UTC [11230] WARNING: there is no transaction in progress
2026-09-25 13:13:17,626 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 13:13:17,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:13:17,842 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 13:13:17,842 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:13:17,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 13:13:18,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 13:13:18,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2d0s.c.gcp-upstream.internal','N') ;"
2026-09-25 13:13:18,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:13:18,792 [I] Table x_portal_user does not exist in database ranger
2026-09-25 13:13:18,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:13:19,018 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 13:13:19,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:13:19,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:13:19,537 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 13:13:19,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 13:13:20.497 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.513 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.529 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.544 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.559 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.577 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.632 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.639 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.653 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.664 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.673 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.686 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.696 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.702 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.708 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:20.715 UTC [11338] WARNING: there is no transaction in progress
2026-09-25 13:13:21,330 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 13:13:21,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 13:13:21,558 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 13:13:21,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:13:21,804 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 13:13:21,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:13:22,070 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 13:13:22,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:13:22,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:13:22,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2d0s.c.gcp-upstream.internal';"
2026-09-25 13:13:22,776 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 13:13:22,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 13:13:23,007 [I] DB_PATCHES have already been applied
I20260925 13:13:23.013527 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpOnpMs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 13:13:23.013592 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 13:13:23.016561 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47221
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 13:13:24.138471 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.249s user 0.000s sys 0.008s
Sep 25, 2026 1:13:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:13:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:13:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:13:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:13:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 13:13:44.934827 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 13:13:45 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342025, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 13:13:45.025740 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 13:13:45.035943 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 13:13:45.825029 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 13:13:45.825119 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms
I20260925 13:13:45.825142 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54173
2026-09-25 13:13:46,000 [I] DB FLAVOR :POSTGRES
2026-09-25 13:13:46,000 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:13:46,000 [I] Checking connection
2026-09-25 13:13:46,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 13:13:46,234 [I] connection success
2026-09-25 13:13:46,235 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 13:13:46,235 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 13:13:46,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 13:13:46,492 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 13:13:46,493 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:13:46,493 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 13:13:46,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 13:13:46,745 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 13:13:47.199462 392 mini_ranger_kms.cc:310] Created kms service
I20260925 13:13:47.369791 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 13:13:47.415992 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 13:13:47 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342025, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:13:47.808985 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 13:13:47.809106 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 13:13:47.809128 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 13:13:47.813040 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11518
I20260925 13:13:47.813156 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46361
Sep 25, 2026 1:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 13:13:49 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:50.018407 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.993s user 0.008s sys 0.151s
I20260925 13:13:50.018517 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 13:13:50.018600 392 mini_ranger_kms.cc:401] 127.0.98.20:46361/kms/v1/keys
Sep 25 13:13:50 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:13:50 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:13:53 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:13:53.050911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42903
--ranger_config_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46361/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:53.133567 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:53.133838 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:53.133862 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:53.135008 11569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:13:53.135048 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:53.135063 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:53.135075 11569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:13:53.135087 11569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:13:53.136660 11569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42903
--ranger_config_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:53.136901 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:53.137145 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:53.142133 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:53.142213 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:53.142161 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:53.142304 11569 server_base.cc:1034] running on GCE node
I20260925 13:13:53.142558 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:53.142881 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:53.144091 11569 hybrid_clock.cc:648] HybridClock initialized: now 1790342033144075 us; error 33 us; skew 500 ppm
Sep 25 13:13:53 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:53.155429 11569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:13:53.156036 11569 webserver.cc:460] Webserver started at http://127.0.98.62:41887/ using document root <none> and password file <none>
I20260925 13:13:53.156296 11569 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:53.156335 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:53.156478 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:13:53 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:13:53.209605 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "76d2745d5872410f9372e206ad3f154b"
format_stamp: "Formatted at 2026-09-25 13:13:53 on dist-test-slave-2d0s"
server_key: "d2aa8fe94e050e2bc5e9363847b27957"
server_key_iv: "0e4361498c7d08e4d5913a105f27af76"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:53.210124 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "76d2745d5872410f9372e206ad3f154b"
format_stamp: "Formatted at 2026-09-25 13:13:53 on dist-test-slave-2d0s"
server_key: "d2aa8fe94e050e2bc5e9363847b27957"
server_key_iv: "0e4361498c7d08e4d5913a105f27af76"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:53.212404 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 13:13:53.224588 11584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:53.224815 11569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:13:53.224922 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "76d2745d5872410f9372e206ad3f154b"
format_stamp: "Formatted at 2026-09-25 13:13:53 on dist-test-slave-2d0s"
server_key: "d2aa8fe94e050e2bc5e9363847b27957"
server_key_iv: "0e4361498c7d08e4d5913a105f27af76"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:53.225042 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:53.259318 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:53.261490 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:53.261667 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:53.266917 11569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42903
I20260925 13:13:53.266940 11636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42903 every 8 connection(s)
I20260925 13:13:53.267330 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:13:53.268000 11637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:53.270650 11637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b: Bootstrap starting.
I20260925 13:13:53.271227 11637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:53.271500 11637 log.cc:824] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:53.272152 11637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b: No bootstrap required, opened a new log
I20260925 13:13:53.274161 11637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d2745d5872410f9372e206ad3f154b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260925 13:13:53.274299 11637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:53.274322 11637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d2745d5872410f9372e206ad3f154b, State: Initialized, Role: FOLLOWER
I20260925 13:13:53.274505 11637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d2745d5872410f9372e206ad3f154b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260925 13:13:53.274572 11637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:13:53.274593 11637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:13:53.274647 11637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:53.275141 11637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d2745d5872410f9372e206ad3f154b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260925 13:13:53.275221 11637 leader_election.cc:302] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d2745d5872410f9372e206ad3f154b; no voters:
I20260925 13:13:53.275429 11637 leader_election.cc:288] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:13:53.275445 11640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:13:53.275556 11640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [term 1 LEADER]: Becoming Leader. State: Replica: 76d2745d5872410f9372e206ad3f154b, State: Running, Role: LEADER
I20260925 13:13:53.275679 11637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:13:53.275696 11640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d2745d5872410f9372e206ad3f154b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260925 13:13:53.276111 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d2745d5872410f9372e206ad3f154b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d2745d5872410f9372e206ad3f154b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } } }
I20260925 13:13:53.276197 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:53.276440 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d2745d5872410f9372e206ad3f154b. Latest consensus state: current_term: 1 leader_uuid: "76d2745d5872410f9372e206ad3f154b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d2745d5872410f9372e206ad3f154b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } } }
I20260925 13:13:53.276727 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b [sys.catalog]: This master's current role is: LEADER
I20260925 13:13:53.277052 11647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:13:53.277450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 11569
I20260925 13:13:53.277523 11647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:13:53.277565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:13:53.277815 392 external_mini_cluster.cc:1428] Setting key f880a5c3642f2401efc31c126d98537d
I20260925 13:13:53.277941 11637 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 13:13:53.279457 11647 catalog_manager.cc:1347] Generated new cluster ID: 62fecac9871d4b4e9ac382c9b22274cd
I20260925 13:13:53.279567 11647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:13:53 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:13:53.293414 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:53.278501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.98.62:42903)
0925 13:13:53.278662 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:53.278672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:53.278701 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:13:53.278821 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:53.278828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:53.278884 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:53.279752 (+ 868us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:53.279858 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:53.281378 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:53.282899 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:53.283520 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:53.283946 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:53.286516 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:53.286521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:53.286521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:53.286534 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:53.288529 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:53.288941 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:53.288945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:53.288945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:53.289039 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:53.289247 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:53.289248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:53.289248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:53.289549 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:53.291843 (+ 2294us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:53.292207 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 13:13:53.292940 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":97,"threads_started":1}
I20260925 13:13:53.296713 11647 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:13:53.297654 11647 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:13:53.300881 11647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b: Generated new TSK 0
I20260925 13:13:53.301106 11647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:13:54.020783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:54.119939 11682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:54.120169 11682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:54.120208 11682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:54.121382 11682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:54.121464 11682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:54.121508 11682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:13:54.123049 11682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:54.123318 11682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:54.123553 11682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:54.129441 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:54.129527 11690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:54.129874 11682 server_base.cc:1034] running on GCE node
W20260925 13:13:54.129444 11688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:54.130225 11682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:54.130501 11682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:54.131705 11682 hybrid_clock.cc:648] HybridClock initialized: now 1790342034131677 us; error 42 us; skew 500 ppm
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:54.134757 11682 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:13:54.135470 11682 webserver.cc:460] Webserver started at http://127.0.98.1:33973/ using document root <none> and password file <none>
I20260925 13:13:54.135710 11682 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:54.135756 11682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:54.135885 11682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:13:54.148693 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d6843b575594cf5a682cdf88a823dd0"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "f80f81cb927acdc6058cd818b1f62532"
server_key_iv: "cb2a44fa367be4275d0d58d950a3c87a"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.149230 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d6843b575594cf5a682cdf88a823dd0"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "f80f81cb927acdc6058cd818b1f62532"
server_key_iv: "cb2a44fa367be4275d0d58d950a3c87a"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.151189 11682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 13:13:54.164512 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.164739 11682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:13:54.164841 11682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "1d6843b575594cf5a682cdf88a823dd0"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "f80f81cb927acdc6058cd818b1f62532"
server_key_iv: "cb2a44fa367be4275d0d58d950a3c87a"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.164944 11682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:54.183549 11682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:54.186405 11682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:54.186563 11682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:54.186834 11682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:54.187288 11682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:54.187391 11682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.187490 11682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:54.187565 11682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.195218 11682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34501
I20260925 13:13:54.195737 11682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:13:54.196878 11812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34501 every 8 connection(s)
I20260925 13:13:54.198148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 11682
I20260925 13:13:54.198261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:13:54.198508 392 external_mini_cluster.cc:1428] Setting key d225abe1b850e7ec2fa6f2329bdc0f18
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:13:54.218585 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.197818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33605 (local address 127.0.98.62:42903)
0925 13:13:54.198613 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.198615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.198626 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:13:54.199598 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.199599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.199612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.199679 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:54.200405 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.201230 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.202794 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.203129 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.203197 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.214956 (+ 11759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:54.214958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:54.214959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:54.214969 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:54.216999 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.217468 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.217470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.217470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.217539 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.217877 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.217878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.217879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.218011 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:54.218087 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.218397 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.218484 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":75,"threads_started":1}
I20260925 13:13:54.218858 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.197586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42903 (local address 127.0.98.1:33605)
0925 13:13:54.198355 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:54.198402 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:54.199475 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:54.199752 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:54.199756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:54.200010 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:54.200288 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.200296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.201337 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.201338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.202687 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.202690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.203271 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.203272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.203417 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.210482 (+ 7065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:54.210498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:54.214795 (+ 4297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:54.217098 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:54.217110 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:54.217113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:54.217376 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:54.217606 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:54.217607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:54.217607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:54.217803 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:54.218078 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:54.218085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:54.218256 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 13:13:54.218366 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 13:13:54.218532 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":329,"threads_started":1}
I20260925 13:13:54.221244 11813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42903
I20260925 13:13:54.221387 11813 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:54.221598 11813 heartbeater.cc:511] Master 127.0.98.62:42903 requested a full tablet report, sending...
I20260925 13:13:54.222268 11601 ts_manager.cc:194] Registered new tserver with Master: 1d6843b575594cf5a682cdf88a823dd0 (127.0.98.1:34501)
I20260925 13:13:54.222942 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:13:54.232117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:54.345453 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:54.345733 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:54.345778 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:54.347002 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:54.347082 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:54.347132 11819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:13:54.348803 11819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:54.349097 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:54.349380 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:54.354576 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:54.354540 11827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:54.355023 11829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:54.355501 11819 server_base.cc:1034] running on GCE node
I20260925 13:13:54.355808 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:54.356102 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:54.357290 11819 hybrid_clock.cc:648] HybridClock initialized: now 1790342034357268 us; error 39 us; skew 500 ppm
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:54.360123 11819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:13:54.360589 11819 webserver.cc:460] Webserver started at http://127.0.98.2:41465/ using document root <none> and password file <none>
I20260925 13:13:54.360841 11819 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:54.360911 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:54.361053 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:54.382771 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "91433fa0aa9546408407e18b9c1a709b"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "38d622840c1464d02bc1957c42a32f92"
server_key_iv: "bad6e6ed6718eb05351b6960f264dd1a"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.383457 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "91433fa0aa9546408407e18b9c1a709b"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "38d622840c1464d02bc1957c42a32f92"
server_key_iv: "bad6e6ed6718eb05351b6960f264dd1a"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.385571 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 13:13:54.408785 11836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.409181 11819 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260925 13:13:54.409282 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "91433fa0aa9546408407e18b9c1a709b"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "38d622840c1464d02bc1957c42a32f92"
server_key_iv: "bad6e6ed6718eb05351b6960f264dd1a"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.409395 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:54.460515 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:54.463232 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:54.463403 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:54.463647 11819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:54.464109 11819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:54.464222 11819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.464308 11819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:54.464370 11819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.472133 11819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41179
I20260925 13:13:54.472734 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:13:54.473899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 11819
I20260925 13:13:54.474015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:13:54.474224 392 external_mini_cluster.cc:1428] Setting key 12fc08ae263e4efa01ebbf56688905b8
I20260925 13:13:54.475602 11949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41179 every 8 connection(s)
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:13:54.486129 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.475743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40013 (local address 127.0.98.62:42903)
0925 13:13:54.475793 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.475796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.476039 (+ 243us) server_negotiation.cc:407] Connection header received
0925 13:13:54.476815 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.476816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.476828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.476913 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:54.477763 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.478552 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.480136 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.480436 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.480546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.482870 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:54.482872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:54.482872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:54.482880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:54.484522 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.484964 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.484966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.484967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.485030 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.485321 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.485322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.485322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.485453 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:54.485530 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.485921 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.486021 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:13:54.486997 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.475746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42903 (local address 127.0.98.2:40013)
0925 13:13:54.475901 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:54.475946 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:54.476691 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:54.477076 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:54.477080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:54.477336 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:54.477624 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.477631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.478667 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.478668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.480009 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.480019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.480558 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.480559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.480715 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.481337 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:54.481345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:54.482765 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:54.484618 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:54.484625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:54.484628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:54.484871 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:54.485110 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:54.485111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:54.485111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:54.485245 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:54.485521 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:54.485528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:54.485753 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 13:13:54.486549 (+ 796us) client_negotiation.cc:240] Negotiation successful
0925 13:13:54.486749 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260925 13:13:54.487526 11950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42903
I20260925 13:13:54.487665 11950 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:54.487874 11950 heartbeater.cc:511] Master 127.0.98.62:42903 requested a full tablet report, sending...
I20260925 13:13:54.488308 11601 ts_manager.cc:194] Registered new tserver with Master: 91433fa0aa9546408407e18b9c1a709b (127.0.98.2:41179)
I20260925 13:13:54.488891 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:13:54.505201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:13:54.599318 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:13:54.599725 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:13:54.599824 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:13:54.601138 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:13:54.601208 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:13:54.601269 11956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:13:54.602941 11956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:13:54.603219 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:13:54.603468 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:13:54.608462 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:54.608444 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:13:54.608808 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:13:54.609316 11956 server_base.cc:1034] running on GCE node
I20260925 13:13:54.609548 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:13:54.609850 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:13:54.611008 11956 hybrid_clock.cc:648] HybridClock initialized: now 1790342034610988 us; error 36 us; skew 500 ppm
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:13:54.613813 11956 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:13:54.614251 11956 webserver.cc:460] Webserver started at http://127.0.98.3:35847/ using document root <none> and password file <none>
I20260925 13:13:54.614509 11956 fs_manager.cc:359] Metadata directory not provided
I20260925 13:13:54.614580 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:13:54.614725 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:13:54.635695 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "744ee5cf8695468090b49f100bf0c02f"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "34ddbb5805019c6b7317e6ffe5c0c160"
server_key_iv: "de94967b42cf490d7d74513d3d81ab5c"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.637135 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "744ee5cf8695468090b49f100bf0c02f"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "34ddbb5805019c6b7317e6ffe5c0c160"
server_key_iv: "de94967b42cf490d7d74513d3d81ab5c"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.639909 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 13:13:54.659138 11971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.660312 11956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 13:13:54.660439 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "744ee5cf8695468090b49f100bf0c02f"
format_stamp: "Formatted at 2026-09-25 13:13:54 on dist-test-slave-2d0s"
server_key: "34ddbb5805019c6b7317e6ffe5c0c160"
server_key_iv: "de94967b42cf490d7d74513d3d81ab5c"
server_key_version: "kuduclusterkey@0"
I20260925 13:13:54.660547 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:13:54.702255 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:13:54.705120 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:13:54.705286 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:13:54.705535 11956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:13:54.705933 11956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:13:54.706007 11956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.706044 11956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:13:54.706071 11956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:13:54.711378 11956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39411
I20260925 13:13:54.711530 12084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39411 every 8 connection(s)
I20260925 13:13:54.711833 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:13:54.712981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 11956
I20260925 13:13:54.713081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:13:54.713296 392 external_mini_cluster.cc:1428] Setting key 1ef791722f2bb641593dccd5cfeaeb4a
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:13:54.723387 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.712910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46391 (local address 127.0.98.62:42903)
0925 13:13:54.712957 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.712960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.713609 (+ 649us) server_negotiation.cc:407] Connection header received
0925 13:13:54.714338 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.714339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.714352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.714381 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:54.715240 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.716356 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.717796 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.717995 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.718059 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.720498 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:54.720499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:54.720500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:54.720506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:54.721993 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.722447 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.722448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.722448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.722500 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.722770 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.722771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.722771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.722867 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:54.722915 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.723250 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.723311 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:13:54.723686 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.712581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42903 (local address 127.0.98.3:46391)
0925 13:13:54.713032 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:54.713081 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:54.714218 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:54.714576 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:54.714581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:54.714840 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:13:54.715134 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.715140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.716473 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.716474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.717706 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.717708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.718084 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.718085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.718232 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.718807 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:13:54.718818 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:13:54.720400 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:13:54.722066 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:54.722074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:54.722076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:54.722370 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:54.722563 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:13:54.722563 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:13:54.722564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:13:54.722707 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:13:54.722928 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:13:54.722933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:13:54.723116 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 13:13:54.723234 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 13:13:54.723392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":79,"threads_started":1}
I20260925 13:13:54.724248 12085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42903
I20260925 13:13:54.724359 12085 heartbeater.cc:464] Registering TS with master...
I20260925 13:13:54.724558 12085 heartbeater.cc:511] Master 127.0.98.62:42903 requested a full tablet report, sending...
I20260925 13:13:54.725307 11601 ts_manager.cc:194] Registered new tserver with Master: 744ee5cf8695468090b49f100bf0c02f (127.0.98.3:39411)
I20260925 13:13:54.725781 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46391
I20260925 13:13:54.735944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:13:54.745304 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.737449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.0.98.62:42903)
0925 13:13:54.737518 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.737520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.737603 (+ 83us) server_negotiation.cc:407] Connection header received
0925 13:13:54.737685 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.737686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.737698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.737757 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:13:54.738448 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.739142 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.740233 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.740514 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.740557 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.742263 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:13:54.742264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:13:54.742265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:13:54.742273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:13:54.743991 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.744396 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.744397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.744398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.744458 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:13:54.744667 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:13:54.744667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:13:54.744668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:13:54.744801 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:13:54.744890 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.745123 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.745200 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:13:54.747866 11601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:13:54.750281 11601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:13:54.766062 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.758485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.0.1:36174)
0925 13:13:54.759004 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:54.759047 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:54.759118 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:54.760493 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:54.760496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:54.760499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:54.760580 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.760582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.761915 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.761916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.763217 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.763220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.765761 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.765762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.765852 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.765869 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 13:13:54.765920 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:13:54.765972 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":114,"threads_started":1}
I20260925 13:13:54.766564 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.760074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.0.98.2:41179)
0925 13:13:54.760282 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.760296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.760307 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:13:54.760348 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.760350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.760366 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.760479 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:54.760661 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.761814 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.764680 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.765626 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.766300 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.766414 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.766443 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.766486 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
I20260925 13:13:54.767234 11884 tablet_service.cc:1467] Processing CreateTablet for tablet 3009ade965c24b47946f69300d5059ab (DEFAULT_TABLE table=test-table [id=e5aa235dcaec4b0985b7007699440574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:13:54.767642 11884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3009ade965c24b47946f69300d5059ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:54.770797 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.758843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.1:34501)
0925 13:13:54.761799 (+ 2956us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.761804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.762001 (+ 197us) server_negotiation.cc:407] Connection header received
0925 13:13:54.762040 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.762042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.762059 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.762135 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:54.762773 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.763797 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.767300 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.768210 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.768587 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.768657 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.770603 (+ 1946us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.770685 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2919,"thread_start_us":156,"threads_started":1}
I20260925 13:13:54.771763 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.758746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34501 (local address 127.0.0.1:33368)
0925 13:13:54.761864 (+ 3118us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:54.761875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:54.761936 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:54.762664 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:54.762665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:54.762667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:54.762694 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.762695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.764015 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.764016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.767184 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.767187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.770363 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.770364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.770452 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.770456 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:13:54.771297 (+ 841us) client_negotiation.cc:240] Negotiation successful
0925 13:13:54.771350 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3101,"thread_start_us":70,"threads_started":1}
I20260925 13:13:54.772637 12103 tablet_bootstrap.cc:492] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Bootstrap starting.
I20260925 13:13:54.774070 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.758486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39411 (local address 127.0.0.1:59482)
0925 13:13:54.760312 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:54.760325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:54.760385 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:54.762529 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:54.762530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:54.762532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:54.762585 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.762587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.768731 (+ 6144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.768732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.770022 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:54.770025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.773345 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.773347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:54.773840 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.773848 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 13:13:54.773892 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:13:54.773962 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1775,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260925 13:13:54.774187 12103 tablet_bootstrap.cc:654] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:54.774493 12103 log.cc:824] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:54.774734 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:54.758780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.0.98.3:39411)
0925 13:13:54.762273 (+ 3493us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:54.762290 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:54.762300 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:13:54.762340 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:54.762342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:54.762357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:54.762440 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:54.764428 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.766997 (+ 2569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.770163 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:54.773222 (+ 3059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:54.774479 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:54.774571 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:54.774604 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 13:13:54.774648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3452,"thread_start_us":89,"threads_started":1}
I20260925 13:13:54.774768 11735 tablet_service.cc:1467] Processing CreateTablet for tablet 3009ade965c24b47946f69300d5059ab (DEFAULT_TABLE table=test-table [id=e5aa235dcaec4b0985b7007699440574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:13:54.775139 11735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3009ade965c24b47946f69300d5059ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:54.777706 12105 tablet_bootstrap.cc:492] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: Bootstrap starting.
I20260925 13:13:54.778424 12105 tablet_bootstrap.cc:654] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:54.778367 12019 tablet_service.cc:1467] Processing CreateTablet for tablet 3009ade965c24b47946f69300d5059ab (DEFAULT_TABLE table=test-table [id=e5aa235dcaec4b0985b7007699440574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:13:54.778687 12019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3009ade965c24b47946f69300d5059ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:13:54.778756 12105 log.cc:824] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:54.779479 12105 tablet_bootstrap.cc:492] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: No bootstrap required, opened a new log
I20260925 13:13:54.779539 12105 ts_tablet_manager.cc:1397] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:13:54.782267 12105 raft_consensus.cc:348] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:54.782469 12107 tablet_bootstrap.cc:492] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Bootstrap starting.
I20260925 13:13:54.782584 12105 raft_consensus.cc:374] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:54.782670 12105 raft_consensus.cc:729] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6843b575594cf5a682cdf88a823dd0, State: Initialized, Role: FOLLOWER
I20260925 13:13:54.782902 12105 consensus_queue.cc:260] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:54.783334 12105 ts_tablet_manager.cc:1428] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260925 13:13:54.783495 11813 heartbeater.cc:503] Master 127.0.98.62:42903 was elected leader, sending a full tablet report...
I20260925 13:13:54.783540 12107 tablet_bootstrap.cc:654] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Neither blocks nor log segments found. Creating new log.
I20260925 13:13:54.783830 12107 log.cc:824] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:13:54.784685 12103 tablet_bootstrap.cc:492] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: No bootstrap required, opened a new log
I20260925 13:13:54.784758 12103 ts_tablet_manager.cc:1397] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260925 13:13:54.786921 12103 raft_consensus.cc:348] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:54.787083 12103 raft_consensus.cc:374] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:54.787127 12103 raft_consensus.cc:729] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91433fa0aa9546408407e18b9c1a709b, State: Initialized, Role: FOLLOWER
I20260925 13:13:54.787262 12103 consensus_queue.cc:260] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:54.787585 11950 heartbeater.cc:503] Master 127.0.98.62:42903 was elected leader, sending a full tablet report...
I20260925 13:13:54.787570 12103 ts_tablet_manager.cc:1428] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:13:54.788870 12107 tablet_bootstrap.cc:492] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: No bootstrap required, opened a new log
I20260925 13:13:54.788957 12107 ts_tablet_manager.cc:1397] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 13:13:54.790413 12107 raft_consensus.cc:348] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:54.790565 12107 raft_consensus.cc:374] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:13:54.790638 12107 raft_consensus.cc:729] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744ee5cf8695468090b49f100bf0c02f, State: Initialized, Role: FOLLOWER
I20260925 13:13:54.790791 12107 consensus_queue.cc:260] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:54.791085 12085 heartbeater.cc:503] Master 127.0.98.62:42903 was elected leader, sending a full tablet report...
I20260925 13:13:54.791071 12107 ts_tablet_manager.cc:1428] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 25 13:13:54 dist-test-slave-2d0s krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 13:13:54.949324 11814 tablet.cc:2369] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:13:54.965679 12086 tablet.cc:2369] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:13:54.975559 11951 tablet.cc:2369] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:13:55.075073 12111 raft_consensus.cc:482] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:13:55.075237 12111 raft_consensus.cc:504] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:55.075680 12111 leader_election.cc:288] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91433fa0aa9546408407e18b9c1a709b (127.0.98.2:41179), 1d6843b575594cf5a682cdf88a823dd0 (127.0.98.1:34501)
I20260925 13:13:55.079916 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:55.075906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.98.3:49371)
0925 13:13:55.076140 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:55.076153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:55.076215 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:55.076398 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:55.076399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:55.076400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:55.076465 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:55.076466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.077474 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.077475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:55.078819 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:55.078821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.079645 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.079646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:55.079722 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:55.079726 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:13:55.079776 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:13:55.079817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":67}
I20260925 13:13:55.080024 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:55.076244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41621 (local address 127.0.98.1:34501)
0925 13:13:55.076286 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:55.076288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:55.076443 (+ 155us) server_negotiation.cc:407] Connection header received
0925 13:13:55.076587 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:55.076588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:55.076600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:55.076630 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:55.076877 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.077572 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.078982 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.079591 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.079649 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:55.079660 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:55.079901 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 13:13:55.079950 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:13:55.080277 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:55.076034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34501 (local address 127.0.98.3:41621)
0925 13:13:55.076343 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:55.076353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:13:55.076472 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:13:55.076697 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:13:55.076698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:13:55.076699 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:13:55.076776 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:55.076778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.077698 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.077698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:55.078875 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:13:55.078878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.079722 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.079723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:13:55.079799 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:55.079803 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:13:55.080143 (+ 340us) client_negotiation.cc:240] Negotiation successful
0925 13:13:55.080185 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260925 13:13:55.080721 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:55.075984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49371 (local address 127.0.98.2:41179)
0925 13:13:55.076042 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:55.076044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:55.076271 (+ 227us) server_negotiation.cc:407] Connection header received
0925 13:13:55.076302 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:55.076302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:55.076312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:55.076354 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:13:55.076547 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.077333 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.078940 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.079528 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.079596 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:55.079608 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:55.080608 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0925 13:13:55.080655 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:13:55.080801 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3009ade965c24b47946f69300d5059ab" candidate_uuid: "744ee5cf8695468090b49f100bf0c02f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6843b575594cf5a682cdf88a823dd0" is_pre_election: true
I20260925 13:13:55.080948 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3009ade965c24b47946f69300d5059ab" candidate_uuid: "744ee5cf8695468090b49f100bf0c02f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91433fa0aa9546408407e18b9c1a709b" is_pre_election: true
I20260925 13:13:55.080988 11767 raft_consensus.cc:2454] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744ee5cf8695468090b49f100bf0c02f in term 0.
I20260925 13:13:55.081077 11900 raft_consensus.cc:2454] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744ee5cf8695468090b49f100bf0c02f in term 0.
I20260925 13:13:55.081228 11974 leader_election.cc:302] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6843b575594cf5a682cdf88a823dd0, 744ee5cf8695468090b49f100bf0c02f; no voters:
I20260925 13:13:55.081432 12111 raft_consensus.cc:2781] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:13:55.081525 12111 raft_consensus.cc:482] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:13:55.081558 12111 raft_consensus.cc:3037] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:55.082492 12111 raft_consensus.cc:504] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:55.082628 12111 leader_election.cc:288] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [CANDIDATE]: Term 1 election: Requested vote from peers 91433fa0aa9546408407e18b9c1a709b (127.0.98.2:41179), 1d6843b575594cf5a682cdf88a823dd0 (127.0.98.1:34501)
I20260925 13:13:55.082844 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3009ade965c24b47946f69300d5059ab" candidate_uuid: "744ee5cf8695468090b49f100bf0c02f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91433fa0aa9546408407e18b9c1a709b"
I20260925 13:13:55.082846 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3009ade965c24b47946f69300d5059ab" candidate_uuid: "744ee5cf8695468090b49f100bf0c02f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6843b575594cf5a682cdf88a823dd0"
I20260925 13:13:55.082914 11900 raft_consensus.cc:3037] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:55.082932 11767 raft_consensus.cc:3037] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:13:55.083760 11900 raft_consensus.cc:2454] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744ee5cf8695468090b49f100bf0c02f in term 1.
I20260925 13:13:55.083871 11767 raft_consensus.cc:2454] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744ee5cf8695468090b49f100bf0c02f in term 1.
I20260925 13:13:55.083922 11974 leader_election.cc:302] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744ee5cf8695468090b49f100bf0c02f, 91433fa0aa9546408407e18b9c1a709b; no voters:
I20260925 13:13:55.084003 12111 raft_consensus.cc:2781] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:13:55.084167 12111 raft_consensus.cc:686] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 1 LEADER]: Becoming Leader. State: Replica: 744ee5cf8695468090b49f100bf0c02f, State: Running, Role: LEADER
I20260925 13:13:55.084260 12111 consensus_queue.cc:237] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } }
I20260925 13:13:55.085052 11600 catalog_manager.cc:5495] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f reported cstate change: term changed from 0 to 1, leader changed from <none> to 744ee5cf8695468090b49f100bf0c02f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "744ee5cf8695468090b49f100bf0c02f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee5cf8695468090b49f100bf0c02f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 } health_report { overall_health: UNKNOWN } } }
I20260925 13:13:55.109635 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:55.106347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.0.98.3:39411)
0925 13:13:55.106471 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 13:13:55.106473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:13:55.106482 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:13:55.106523 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:13:55.106524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:13:55.106550 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:13:55.106629 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:13:55.106827 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.107551 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.108668 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:13:55.108977 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:13:55.109058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:13:55.109243 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:13:55.109388 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:13:55.109479 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 13:13:55.109532 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 13:13:55.109556 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 13:13:55.111924 11900 raft_consensus.cc:1261] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 1 FOLLOWER]: Refusing update from remote peer 744ee5cf8695468090b49f100bf0c02f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:13:55.112291 11767 raft_consensus.cc:1261] T 3009ade965c24b47946f69300d5059ab P 1d6843b575594cf5a682cdf88a823dd0 [term 1 FOLLOWER]: Refusing update from remote peer 744ee5cf8695468090b49f100bf0c02f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:13:55.112468 12111 consensus_queue.cc:1035] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91433fa0aa9546408407e18b9c1a709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:13:55.112596 12113 consensus_queue.cc:1035] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6843b575594cf5a682cdf88a823dd0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:13:55.115942 12116 mvcc.cc:205] Tried to move back new op lower bound from 7333240975815180288 to 7333240975705620480. Current Snapshot: MvccSnapshot[applied={T|T < 7333240975815180288}]
I20260925 13:13:55.116317 12117 mvcc.cc:205] Tried to move back new op lower bound from 7333240975815180288 to 7333240975705620480. Current Snapshot: MvccSnapshot[applied={T|T < 7333240975815180288}]
I20260925 13:13:55.118171 12118 mvcc.cc:205] Tried to move back new op lower bound from 7333240975815180288 to 7333240975705620480. Current Snapshot: MvccSnapshot[applied={T|T < 7333240975815180288}]
I20260925 13:13:55.118835 11601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:13:55.118949 11601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:13:55.119746 11601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76d2745d5872410f9372e206ad3f154b: Sending DeleteTablet for 3 replicas of tablet 3009ade965c24b47946f69300d5059ab
I20260925 13:13:55.120244 12019 tablet_service.cc:1514] Processing DeleteTablet for tablet 3009ade965c24b47946f69300d5059ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:13:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59482
I20260925 13:13:55.120437 11884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3009ade965c24b47946f69300d5059ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:13:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36174
I20260925 13:13:55.120636 11735 tablet_service.cc:1514] Processing DeleteTablet for tablet 3009ade965c24b47946f69300d5059ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:13:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33368
I20260925 13:13:55.121245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 11682
I20260925 13:13:55.125958 12125 tablet_replica.cc:330] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: stopping tablet replica
I20260925 13:13:55.126081 12126 tablet_replica.cc:330] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: stopping tablet replica
I20260925 13:13:55.126259 12125 raft_consensus.cc:2229] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:13:55.126403 12126 raft_consensus.cc:2229] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:13:55.126600 12125 raft_consensus.cc:2258] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:13:55.126780 12126 raft_consensus.cc:2258] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:13:55.127348 12126 ts_tablet_manager.cc:1905] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:13:55.133363 12126 ts_tablet_manager.cc:1918] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:13:55.133553 12126 log.cc:1196] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/3009ade965c24b47946f69300d5059ab
I20260925 13:13:55.133936 12126 ts_tablet_manager.cc:1939] T 3009ade965c24b47946f69300d5059ab P 91433fa0aa9546408407e18b9c1a709b: Deleting consensus metadata
I20260925 13:13:55.134522 11587 catalog_manager.cc:4841] TS 91433fa0aa9546408407e18b9c1a709b (127.0.98.2:41179): tablet 3009ade965c24b47946f69300d5059ab (table test-table [id=e5aa235dcaec4b0985b7007699440574]) successfully deleted
W20260925 13:13:55.134927 11587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34501 (error 108)
I20260925 13:13:55.135577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 11819
I20260925 13:13:55.137284 12125 ts_tablet_manager.cc:1905] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:13:55.139389 12125 ts_tablet_manager.cc:1918] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:13:55.139477 12125 log.cc:1196] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/3009ade965c24b47946f69300d5059ab
I20260925 13:13:55.139740 12125 ts_tablet_manager.cc:1939] T 3009ade965c24b47946f69300d5059ab P 744ee5cf8695468090b49f100bf0c02f: Deleting consensus metadata
I20260925 13:13:55.140234 11588 catalog_manager.cc:4841] TS 744ee5cf8695468090b49f100bf0c02f (127.0.98.3:39411): tablet 3009ade965c24b47946f69300d5059ab (table test-table [id=e5aa235dcaec4b0985b7007699440574]) successfully deleted
I20260925 13:13:55.142735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 11956
I20260925 13:13:55.144032 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:13:55.143858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34501 (local address 127.0.0.1:33370)
0925 13:13:55.143935 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:13:55.143992 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 13:13:55.144217 11587 catalog_manager.cc:4584] TS 1d6843b575594cf5a682cdf88a823dd0 (127.0.98.1:34501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3009ade965c24b47946f69300d5059ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34501: connect: Connection refused (error 111)
I20260925 13:13:55.149034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 11569
I20260925 13:13:55.155792 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 13:13:55.500746 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 13:13:55.500880 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 13:13:55.846140 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 13:13:55.846 UTC [11155] LOG: received smart shutdown request
2026-09-25 13:13:55.849 UTC [11155] LOG: background worker "logical replication launcher" (PID 11164) exited with exit code 1
2026-09-25 13:13:55.850 UTC [11159] LOG: shutting down
2026-09-25 13:13:56.024 UTC [11155] LOG: database system is shut down
2026-09-25T13:13:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45295 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:13:56 dist-test-slave-2d0s krb5kdc[12137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:13:56 dist-test-slave-2d0s krb5kdc[12137](info): set up 2 sockets
Sep 25 13:13:56 dist-test-slave-2d0s krb5kdc[12137](info): commencing operation
krb5kdc: starting...
W20260925 13:13:57.161528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:13:57 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:13:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 13:13:57.324398 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpOnpMs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/postgres -l logfile start
2026-09-25 13:13:59.981 UTC [12167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 13:13:59.982 UTC [12167] LOG: listening on IPv4 address "127.0.98.20", port 36725
2026-09-25 13:13:59.985 UTC [12167] LOG: listening on Unix socket "/tmp/.s.PGSQL.36725"
2026-09-25 13:14:00.025 UTC [12170] LOG: database system was shut down at 2026-09-25 13:13:58 UTC
2026-09-25 13:14:00.030 UTC [12167] LOG: database system is ready to accept connections
I20260925 13:14:00.953253 392 mini_postgres.cc:96] Postgres bound to 36725
2026-09-25 13:14:00.959 UTC [12178] FATAL: database "slave" does not exist
127.0.98.20:36725 - accepting connections
I20260925 13:14:00.959669 392 mini_ranger.cc:161] Starting Ranger...
I20260925 13:14:00.974545 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 13:14:01.438995 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 13:14:01.439082 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-admin
2026-09-25 13:14:01,732 [I] DB FLAVOR :POSTGRES
2026-09-25 13:14:01,733 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:14:01,733 [I] Checking connection..
2026-09-25 13:14:01,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 13:14:01,963 [I] Checking connection passed.
2026-09-25 13:14:01,964 [I] --------- Verifying version history table ---------
2026-09-25 13:14:01,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:14:02,226 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:14:02,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:14:02,469 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:14:02,469 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 13:14:02,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 13:14:02.731 UTC [12242] WARNING: there is no transaction in progress
2026-09-25 13:14:02,737 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 13:14:02,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:14:02,998 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 13:14:02,998 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:14:02,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 13:14:03,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 13:14:03,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2d0s.c.gcp-upstream.internal','N') ;"
2026-09-25 13:14:03,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:14:04,061 [I] Table x_portal_user does not exist in database ranger
2026-09-25 13:14:04,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:14:04,311 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 13:14:04,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:14:04,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:14:04,782 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 13:14:04,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 13:14:05.715 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.730 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.746 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.761 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.776 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.793 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.847 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.853 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.867 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.877 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.887 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.900 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.909 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.915 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.921 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:05.928 UTC [12350] WARNING: there is no transaction in progress
2026-09-25 13:14:06,520 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 13:14:06,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 13:14:06,745 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 13:14:06,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:14:07,005 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 13:14:07,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:14:07,238 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 13:14:07,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:14:07,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:14:07,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2d0s.c.gcp-upstream.internal';"
2026-09-25 13:14:07,958 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 13:14:07,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 13:14:08,190 [I] DB_PATCHES have already been applied
I20260925 13:14:08.195902 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpOnpMs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 13:14:08.195974 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 13:14:08.198870 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52615
Sep 25, 2026 1:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:14:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:14:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 13:14:09.287150 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.327s user 0.000s sys 0.008s
Sep 25, 2026 1:14:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:14:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 1:14:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:14:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:14:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 13:14:31.295480 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 13:14:31 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342071, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 13:14:31.379052 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 13:14:31.388718 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 13:14:32.250276 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 13:14:32.250408 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms
I20260925 13:14:32.250429 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36725
2026-09-25 13:14:32,411 [I] DB FLAVOR :POSTGRES
2026-09-25 13:14:32,411 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:14:32,411 [I] Checking connection
2026-09-25 13:14:32,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 13:14:32,674 [I] connection success
2026-09-25 13:14:32,674 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 13:14:32,674 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 13:14:32,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 13:14:32,917 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 13:14:32,917 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:14:32,917 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 13:14:32,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 13:14:33,182 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 13:14:33.696941 392 mini_ranger_kms.cc:310] Created kms service
I20260925 13:14:33.888152 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 13:14:33.936972 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 13:14:33 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342071, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:34.348332 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 13:14:34.348555 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpOnpMs/build/release/bin/postgresql.jar:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpOnpMs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 13:14:34.348577 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 13:14:34.351934 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12530
I20260925 13:14:34.352020 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46357
Sep 25, 2026 1:14:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:14:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:14:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:14:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:14:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:14:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 13:14:36.564080 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.185s user 0.002s sys 0.141s
I20260925 13:14:36.564188 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 13:14:36.564244 392 mini_ranger_kms.cc:401] 127.0.98.20:46357/kms/v1/keys
Sep 25 13:14:36 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 13:14:37 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:14:37 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:14:39 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:14:39.665766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34041
--ranger_config_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:39.753275 12581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:39.753485 12581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:39.753506 12581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:39.754705 12581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:39.754750 12581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:39.754763 12581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:39.754776 12581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:39.754788 12581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:39.756423 12581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34041
--ranger_config_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:39.756670 12581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:39.756884 12581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:39.762462 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:39.762658 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:39.762953 12581 server_base.cc:1034] running on GCE node
W20260925 13:14:39.762665 12589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:39.763254 12581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:39.763574 12581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:39.764765 12581 hybrid_clock.cc:648] HybridClock initialized: now 1790342079764718 us; error 42 us; skew 500 ppm
Sep 25 13:14:39 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:39.771616 12581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:39.772104 12581 webserver.cc:460] Webserver started at http://127.0.98.62:38187/ using document root <none> and password file <none>
I20260925 13:14:39.772341 12581 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:39.772400 12581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:39.772569 12581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:14:39 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:39.831032 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "f09672174a07441e8341d71c8cb688db"
format_stamp: "Formatted at 2026-09-25 13:14:39 on dist-test-slave-2d0s"
server_key: "f3c1f71538adc171b5d865950d27530d"
server_key_iv: "73c7f85b0b6b64d65f2aca118713561f"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:39.831382 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "f09672174a07441e8341d71c8cb688db"
format_stamp: "Formatted at 2026-09-25 13:14:39 on dist-test-slave-2d0s"
server_key: "f3c1f71538adc171b5d865950d27530d"
server_key_iv: "73c7f85b0b6b64d65f2aca118713561f"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:39.832902 12581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:14:39.847945 12596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:39.848150 12581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:14:39.848272 12581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "f09672174a07441e8341d71c8cb688db"
format_stamp: "Formatted at 2026-09-25 13:14:39 on dist-test-slave-2d0s"
server_key: "f3c1f71538adc171b5d865950d27530d"
server_key_iv: "73c7f85b0b6b64d65f2aca118713561f"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:39.848382 12581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:39.862512 12581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:39.864465 12581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:39.864619 12581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:39.869563 12581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34041
I20260925 13:14:39.869645 12648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34041 every 8 connection(s)
I20260925 13:14:39.870044 12581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:39.870754 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:39.872255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 12581
I20260925 13:14:39.872375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:14:39.872601 392 external_mini_cluster.cc:1428] Setting key d9ebdd3f1287eb5b9ff24fbf270d7927
I20260925 13:14:39.873515 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Bootstrap starting.
I20260925 13:14:39.874176 12649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:39.874519 12649 log.cc:824] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:39.875237 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: No bootstrap required, opened a new log
I20260925 13:14:39.877475 12649 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:39.877640 12649 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:39.877708 12649 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09672174a07441e8341d71c8cb688db, State: Initialized, Role: FOLLOWER
I20260925 13:14:39.877883 12649 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:39.877978 12649 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:14:39.878031 12649 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:14:39.878085 12649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:39.878698 12649 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:39.878813 12649 leader_election.cc:302] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09672174a07441e8341d71c8cb688db; no voters:
I20260925 13:14:39.879041 12649 leader_election.cc:288] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:14:39.879101 12654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:39.879340 12654 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 LEADER]: Becoming Leader. State: Replica: f09672174a07441e8341d71c8cb688db, State: Running, Role: LEADER
I20260925 13:14:39.879457 12649 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:39.879477 12654 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:39.879922 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09672174a07441e8341d71c8cb688db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } } }
I20260925 13:14:39.880077 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:39.880342 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09672174a07441e8341d71c8cb688db. Latest consensus state: current_term: 1 leader_uuid: "f09672174a07441e8341d71c8cb688db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } } }
I20260925 13:14:39.880492 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:39.880656 12663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:39.881160 12663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 13:14:39 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:39.881537 12649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 13:14:39.891402 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:39.873199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.0.98.62:34041)
0925 13:14:39.873397 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:39.873407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:39.873440 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:14:39.874043 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:39.874052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:39.874123 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:39.874299 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:39.875252 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:39.876728 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:39.877862 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:39.878501 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:39.878628 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:39.883641 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:39.883714 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:39.883714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:39.883836 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:39.889250 (+ 5414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:39.889814 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:39.889818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:39.889818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:39.889922 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:39.890171 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:39.890172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:39.890172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:39.890518 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:39.890668 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:39.890968 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 13:14:39.891136 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260925 13:14:39.897815 12663 catalog_manager.cc:1347] Generated new cluster ID: da1b36102a654e2890a5b78986b6c938
I20260925 13:14:39.897887 12663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:14:39.922894 12663 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:14:39.923736 12663 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:14:39.929148 12663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Generated new TSK 0
I20260925 13:14:39.929468 12663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:14:40.698277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34041
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:40.818512 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:40.818809 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:40.818861 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:40.820590 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:40.820655 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:40.820698 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:14:40.822909 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:40.823171 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:40.823431 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:40.828722 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:40.828944 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:40.829128 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:40.829151 12694 server_base.cc:1034] running on GCE node
I20260925 13:14:40.829537 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:40.829845 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:40.831085 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790342080831056 us; error 44 us; skew 500 ppm
Sep 25 13:14:40 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:40.834275 12694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:14:40.834832 12694 webserver.cc:460] Webserver started at http://127.0.98.1:34651/ using document root <none> and password file <none>
I20260925 13:14:40.835065 12694 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:40.835132 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:40.835278 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:14:40 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:40.850072 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "854548e4771b4c5c9c52e74f73eab2aa"
format_stamp: "Formatted at 2026-09-25 13:14:40 on dist-test-slave-2d0s"
server_key: "d70254fd7f4965cb6a90ada0d9d482c8"
server_key_iv: "82e3cfb41722c70172aed3791307906d"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:40.850625 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "854548e4771b4c5c9c52e74f73eab2aa"
format_stamp: "Formatted at 2026-09-25 13:14:40 on dist-test-slave-2d0s"
server_key: "d70254fd7f4965cb6a90ada0d9d482c8"
server_key_iv: "82e3cfb41722c70172aed3791307906d"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:40.852766 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:40.877808 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:40.878108 12694 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.000s
I20260925 13:14:40.878234 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "854548e4771b4c5c9c52e74f73eab2aa"
format_stamp: "Formatted at 2026-09-25 13:14:40 on dist-test-slave-2d0s"
server_key: "d70254fd7f4965cb6a90ada0d9d482c8"
server_key_iv: "82e3cfb41722c70172aed3791307906d"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:40.878361 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:40.897003 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:40.899521 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:40.899690 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:40.899974 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:40.900334 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:40.900380 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:40.900411 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:40.900429 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:40.907935 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33345
I20260925 13:14:40.908488 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:14:40.909511 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33345 every 8 connection(s)
I20260925 13:14:40.914948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 12694
I20260925 13:14:40.915060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:14:40.915282 392 external_mini_cluster.cc:1428] Setting key fd287ed755634fe140ba878af3fea8e2
Sep 25 13:14:40 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:40.920109 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:40.909249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34041 (local address 127.0.98.1:52801)
0925 13:14:40.909502 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:40.909544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:40.910229 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:40.910586 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:40.910590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:40.910860 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:40.911130 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:40.911136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:40.912179 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:40.912180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:40.913461 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:40.913464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:40.914020 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:40.914021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:40.914165 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:40.914820 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:40.914830 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:40.916440 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:40.918515 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:40.918521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:40.918524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:40.918811 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:40.919010 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:40.919011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:40.919011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:40.919128 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:40.919422 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:40.919427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:40.919597 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 13:14:40.919703 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:14:40.919875 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260925 13:14:40.920575 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:40.909253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52801 (local address 127.0.98.62:34041)
0925 13:14:40.909914 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:40.909916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:40.909928 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:14:40.910367 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:40.910368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:40.910383 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:40.910492 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:40.911250 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:40.912067 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:40.913564 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:40.913897 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:40.913972 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:40.916543 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:40.916544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:40.916545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:40.916552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:40.918414 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:40.918891 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:40.918892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:40.918892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:40.918951 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:40.919210 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:40.919210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:40.919211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:40.919343 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:40.919399 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:40.920433 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0925 13:14:40.920500 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":76,"threads_started":1}
I20260925 13:14:40.921144 12825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34041
I20260925 13:14:40.921240 12825 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:40.921406 12825 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:40.921909 12613 ts_manager.cc:194] Registered new tserver with Master: 854548e4771b4c5c9c52e74f73eab2aa (127.0.98.1:33345)
I20260925 13:14:40.922529 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52801
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:14:40.945421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34041
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:41.055840 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:41.056082 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:41.056121 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:41.057333 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:41.057407 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:41.057461 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:14:41.059168 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:41.059443 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:41.059706 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:41.064754 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.064728 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.065088 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:41.065232 12831 server_base.cc:1034] running on GCE node
I20260925 13:14:41.065534 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:41.065842 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:41.067008 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790342081066989 us; error 38 us; skew 500 ppm
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:41.070169 12831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:14:41.070734 12831 webserver.cc:460] Webserver started at http://127.0.98.2:46313/ using document root <none> and password file <none>
I20260925 13:14:41.070992 12831 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:41.071063 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:41.071257 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:41.095965 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "d63c2a70ca204e2a97950e4d36c1ab7d"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "d99a9594795486c186076fca0581aff5"
server_key_iv: "2c1b3b8344130b00a7a18adffba1f0d4"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.096550 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d63c2a70ca204e2a97950e4d36c1ab7d"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "d99a9594795486c186076fca0581aff5"
server_key_iv: "2c1b3b8344130b00a7a18adffba1f0d4"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.099345 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:41.123088 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.123329 12831 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260925 13:14:41.123454 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "d63c2a70ca204e2a97950e4d36c1ab7d"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "d99a9594795486c186076fca0581aff5"
server_key_iv: "2c1b3b8344130b00a7a18adffba1f0d4"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.123548 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:41.135782 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:41.138321 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:41.138520 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:41.138787 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:41.139159 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:41.139240 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.139305 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:41.139360 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.146286 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38197
I20260925 13:14:41.146449 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38197 every 8 connection(s)
I20260925 13:14:41.146780 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:41.156975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 12831
I20260925 13:14:41.157091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:14:41.157281 392 external_mini_cluster.cc:1428] Setting key f3b0bfbe537eacebac2d45e02fab85df
I20260925 13:14:41.158540 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:41.147760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46597 (local address 127.0.98.62:34041)
0925 13:14:41.147811 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:41.147813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:41.147830 (+ 17us) server_negotiation.cc:407] Connection header received
0925 13:14:41.148533 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:41.148534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:41.148546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:41.148612 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:41.149310 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.150095 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.151639 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.151975 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.152046 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:41.154646 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:41.154648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:41.154648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:41.154656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:41.156345 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:41.156862 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:41.156863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:41.156863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:41.156920 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:41.157203 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:41.157204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:41.157204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:41.157336 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:41.157384 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:41.158338 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 13:14:41.158448 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:14:41.159250 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:41.147513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34041 (local address 127.0.98.2:46597)
0925 13:14:41.147704 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:41.147758 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:41.148392 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:41.148690 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:41.148695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:41.148923 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:41.149199 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:41.149204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.150201 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.150202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:41.151510 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:41.151513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.152063 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.152064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:41.152203 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:41.152818 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:41.152828 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:41.154528 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:41.156453 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:41.156459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:41.156461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:41.156757 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:41.156994 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:41.156995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:41.156995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:41.157124 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:41.157995 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:41.158002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:41.158175 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 13:14:41.158867 (+ 692us) client_negotiation.cc:240] Negotiation successful
0925 13:14:41.159033 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260925 13:14:41.159930 12962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34041
I20260925 13:14:41.160024 12962 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:41.160212 12962 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:41.160692 12613 ts_manager.cc:194] Registered new tserver with Master: d63c2a70ca204e2a97950e4d36c1ab7d (127.0.98.2:38197)
I20260925 13:14:41.161154 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46597
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:14:41.194377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34041
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:41.312830 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:41.313172 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:41.313251 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:41.314956 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:41.315083 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:41.315176 12968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:14:41.317322 12968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:41.317880 12968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:41.318202 12968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:41.324536 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.324680 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.324504 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:41.325870 12968 server_base.cc:1034] running on GCE node
I20260925 13:14:41.326172 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:41.326455 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:41.333758 12968 hybrid_clock.cc:648] HybridClock initialized: now 1790342081333742 us; error 63 us; skew 500 ppm
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:41.337054 12968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:14:41.337581 12968 webserver.cc:460] Webserver started at http://127.0.98.3:41831/ using document root <none> and password file <none>
I20260925 13:14:41.337946 12968 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:41.338066 12968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:41.338233 12968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:41.355007 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f9d2b286d22467486b231222a4e72b1"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "be9f19f8174f038a110d39f0b6277210"
server_key_iv: "076b56b34a1f9bf7c7dd5e23a85504c8"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.355527 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f9d2b286d22467486b231222a4e72b1"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "be9f19f8174f038a110d39f0b6277210"
server_key_iv: "076b56b34a1f9bf7c7dd5e23a85504c8"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.357390 12968 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:41.369506 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.369786 12968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:41.369899 12968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "5f9d2b286d22467486b231222a4e72b1"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "be9f19f8174f038a110d39f0b6277210"
server_key_iv: "076b56b34a1f9bf7c7dd5e23a85504c8"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.370002 12968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:41.406783 12968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:41.409395 12968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:41.409611 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:41.409945 12968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:41.410365 12968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:41.410456 12968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.410538 12968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:41.410600 12968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.416765 12968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41727
I20260925 13:14:41.417261 12968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:14:41.419217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 12968
I20260925 13:14:41.419327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:14:41.419516 392 external_mini_cluster.cc:1428] Setting key 94b533d23d6529a03b2713da9c0d583a
I20260925 13:14:41.421311 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41727 every 8 connection(s)
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:41.439658 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:41.422228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60697 (local address 127.0.98.62:34041)
0925 13:14:41.422277 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:41.422279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:41.422445 (+ 166us) server_negotiation.cc:407] Connection header received
0925 13:14:41.424048 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:41.424050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:41.424063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:41.424092 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:41.424970 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.425701 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.428081 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.428370 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.428412 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:41.435726 (+ 7314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:41.435727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:41.435728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:41.435736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:41.437526 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:41.438459 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:41.438461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:41.438461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:41.438530 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:41.438832 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:41.438833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:41.438833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:41.438987 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:41.439032 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:41.439464 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 13:14:41.439555 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:14:41.440205 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:41.421956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34041 (local address 127.0.98.3:60697)
0925 13:14:41.422312 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:41.422367 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:41.423904 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:41.424289 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:41.424294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:41.424573 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:41.424852 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:41.424857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.426175 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.426176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:41.427978 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:41.427981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.428521 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.428522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:41.428660 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:41.429287 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:41.429298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:41.435604 (+ 6306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:41.437949 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:41.437960 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:41.437964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:41.438352 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:41.438605 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:41.438606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:41.438606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:41.438750 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:41.439082 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:41.439088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:41.439287 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 13:14:41.439776 (+ 489us) client_negotiation.cc:240] Negotiation successful
0925 13:14:41.439961 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260925 13:14:41.441155 13097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34041
I20260925 13:14:41.441259 13097 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:41.441473 13097 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:41.441864 12613 ts_manager.cc:194] Registered new tserver with Master: 5f9d2b286d22467486b231222a4e72b1 (127.0.98.3:41727)
I20260925 13:14:41.442351 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60697
I20260925 13:14:41.443703 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:14:41.444363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 12694
I20260925 13:14:41.452072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 12831
I20260925 13:14:41.457985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 12968
I20260925 13:14:41.463169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 12581
I20260925 13:14:41.470181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34041
--webserver_interface=127.0.98.62
--webserver_port=38187
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34041
--ranger_config_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:41.552418 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:41.552649 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:41.552672 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:41.553860 13100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:41.553903 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:41.553917 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:41.553930 13100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:41.553943 13100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:41.555522 13100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34041
--ranger_config_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:41.555759 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:41.555992 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:41.560873 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.560904 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.560930 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:41.561100 13100 server_base.cc:1034] running on GCE node
I20260925 13:14:41.561374 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:41.561723 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:41.562884 13100 hybrid_clock.cc:648] HybridClock initialized: now 1790342081562870 us; error 42 us; skew 500 ppm
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:41.565521 13100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:41.566026 13100 webserver.cc:460] Webserver started at http://127.0.98.62:38187/ using document root <none> and password file <none>
I20260925 13:14:41.566262 13100 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:41.566344 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:41.627965 13100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:41.628819 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.629000 13100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:41.629117 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "f09672174a07441e8341d71c8cb688db"
format_stamp: "Formatted at 2026-09-25 13:14:39 on dist-test-slave-2d0s"
server_key: "f3c1f71538adc171b5d865950d27530d"
server_key_iv: "73c7f85b0b6b64d65f2aca118713561f"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.629499 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:41.645292 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:41.647435 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:41.647603 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:41.652321 13100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34041
I20260925 13:14:41.652403 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34041 every 8 connection(s)
I20260925 13:14:41.652778 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:41.654438 13169 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:14:41.655299 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Bootstrap starting.
I20260925 13:14:41.656816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13100
I20260925 13:14:41.657099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33345
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34041
--webserver_port=34651
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:41.665465 13169 log.cc:824] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:41.667961 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:14:41.668319 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Bootstrap complete.
I20260925 13:14:41.672128 13169 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:41.672454 13169 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09672174a07441e8341d71c8cb688db, State: Initialized, Role: FOLLOWER
I20260925 13:14:41.672633 13169 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:41.672699 13169 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:14:41.672744 13169 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:14:41.672788 13169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:14:41.674113 13169 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:41.674226 13169 leader_election.cc:302] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09672174a07441e8341d71c8cb688db; no voters:
I20260925 13:14:41.674401 13169 leader_election.cc:288] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:14:41.674479 13173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:14:41.674626 13169 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:41.674667 13173 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [term 2 LEADER]: Becoming Leader. State: Replica: f09672174a07441e8341d71c8cb688db, State: Running, Role: LEADER
I20260925 13:14:41.674868 13173 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } }
I20260925 13:14:41.675168 13174 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f09672174a07441e8341d71c8cb688db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } } }
I20260925 13:14:41.675241 13174 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:41.675599 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09672174a07441e8341d71c8cb688db. Latest consensus state: current_term: 2 leader_uuid: "f09672174a07441e8341d71c8cb688db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09672174a07441e8341d71c8cb688db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34041 } } }
I20260925 13:14:41.675666 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:41.676079 13185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:41.676565 13185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:14:41.676685 13185 catalog_manager.cc:1259] Loaded cluster ID: da1b36102a654e2890a5b78986b6c938
I20260925 13:14:41.676718 13185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:14:41.676913 13169 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 13:14:41.680346 13185 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:14:41.680928 13185 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Loaded TSK: 0
I20260925 13:14:41.681234 13185 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:14:41.750479 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:41.750705 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:41.750746 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:41.752405 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:41.752568 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:41.752655 13171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:14:41.754925 13171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34651
--tserver_master_addrs=127.0.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:41.755180 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:41.755488 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:41.760931 13208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.761258 13207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:41.761487 13171 server_base.cc:1034] running on GCE node
W20260925 13:14:41.761740 13210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:41.762008 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:41.762290 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:41.763445 13171 hybrid_clock.cc:648] HybridClock initialized: now 1790342081763428 us; error 35 us; skew 500 ppm
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:41.766185 13171 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:14:41.766727 13171 webserver.cc:460] Webserver started at http://127.0.98.1:34651/ using document root <none> and password file <none>
I20260925 13:14:41.766968 13171 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:41.767052 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:41.786111 13171 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 13:14:41.789811 13218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.790002 13171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 13:14:41.790113 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "854548e4771b4c5c9c52e74f73eab2aa"
format_stamp: "Formatted at 2026-09-25 13:14:40 on dist-test-slave-2d0s"
server_key: "d70254fd7f4965cb6a90ada0d9d482c8"
server_key_iv: "82e3cfb41722c70172aed3791307906d"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.790580 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:41.801558 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:41.803699 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:41.803864 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:41.804118 13171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:41.804484 13171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:41.804565 13171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.804615 13171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:41.804647 13171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.811982 13171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33345
I20260925 13:14:41.812463 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:14:41.814929 13331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33345 every 8 connection(s)
I20260925 13:14:41.818424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13171
I20260925 13:14:41.818729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38197
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34041
--webserver_port=46313
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:41.828488 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:41.813302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34041 (local address 127.0.98.1:44383)
0925 13:14:41.814054 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:41.814102 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:41.814611 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:41.815894 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:41.815897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:41.816138 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:41.816441 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:41.816447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.818149 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.818150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:41.819483 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:41.819486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.820471 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.820473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:41.820599 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:41.822171 (+ 1572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:41.822182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:41.823808 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:41.826377 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:41.826383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:41.826386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:41.826713 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:41.827011 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:41.827012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:41.827013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:41.827153 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:41.827696 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:41.827701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:41.827884 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 13:14:41.828005 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 13:14:41.828180 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":80,"threads_started":1}
I20260925 13:14:41.829255 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:41.813697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44383 (local address 127.0.98.62:34041)
0925 13:14:41.814090 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:41.814102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:41.814195 (+ 93us) server_negotiation.cc:407] Connection header received
0925 13:14:41.815648 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:41.815654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:41.815698 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:41.815853 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:41.816583 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.818026 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.819580 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:41.820250 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:41.820366 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:41.823947 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:41.823951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:41.823951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:41.823964 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:41.826264 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:41.826809 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:41.826813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:41.826814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:41.826929 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:41.827230 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:41.827231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:41.827231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:41.827579 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:41.827665 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:41.828843 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0925 13:14:41.828967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":98,"threads_started":1}
I20260925 13:14:41.830081 13332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34041
I20260925 13:14:41.830180 13332 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:41.830353 13332 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
W20260925 13:14:41.831207 13332 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34041 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 13:14:41.831293 13332 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:41.831362 13332 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:41.831722 13332 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:41.831806 13332 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
W20260925 13:14:41.832141 13332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:14:41.942018 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:41.942303 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:41.942341 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:41.944105 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:41.944167 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:41.944209 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:14:41.946381 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46313
--tserver_master_addrs=127.0.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:41.946806 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:41.947132 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:41.954061 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.954227 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:41.954152 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:41.954909 13336 server_base.cc:1034] running on GCE node
I20260925 13:14:41.955125 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:41.955392 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:41.956604 13336 hybrid_clock.cc:648] HybridClock initialized: now 1790342081956588 us; error 32 us; skew 500 ppm
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:41.959563 13336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:14:41.960070 13336 webserver.cc:460] Webserver started at http://127.0.98.2:46313/ using document root <none> and password file <none>
I20260925 13:14:41.960304 13336 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:41.960395 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:14:41 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:41.986212 13336 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 13:14:41.987177 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:41.987387 13336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:41.987475 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "d63c2a70ca204e2a97950e4d36c1ab7d"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "d99a9594795486c186076fca0581aff5"
server_key_iv: "2c1b3b8344130b00a7a18adffba1f0d4"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:41.987946 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:42.008342 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:42.011181 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:42.011453 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:42.011791 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:42.012212 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:42.012326 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:42.012410 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:42.012478 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:42.019709 13336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38197
I20260925 13:14:42.020373 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:14:42.021387 13465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38197 every 8 connection(s)
Sep 25 13:14:42 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:42.030246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13336
I20260925 13:14:42.030560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41727
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34041
--webserver_port=41831
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:42.032836 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:42.022411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51941 (local address 127.0.98.62:34041)
0925 13:14:42.022468 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:42.022470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:42.022696 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:14:42.023483 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:42.023485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:42.023502 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:42.023539 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:42.024457 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.025335 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.026627 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.026860 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.026962 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:42.029177 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:42.029179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:42.029179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:42.029187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:42.031044 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:42.031529 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:42.031530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:42.031531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:42.031605 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:42.031953 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:42.031954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:42.031954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:42.032118 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:42.032200 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:42.032611 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 13:14:42.032714 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:14:42.033027 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:42.022131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34041 (local address 127.0.98.2:51941)
0925 13:14:42.022557 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:42.022604 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:42.023340 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:42.023741 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:42.023746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:42.024021 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:42.024331 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:42.024339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.025465 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.025467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:42.026506 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:42.026509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.026986 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.026987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:42.027111 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:42.027728 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:42.027737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:42.029057 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:42.031149 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:42.031156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:42.031159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:42.031407 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:42.031691 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:42.031693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:42.031694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:42.031867 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:42.032205 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:42.032213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:42.032411 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 13:14:42.032554 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 13:14:42.032786 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":92,"threads_started":1}
I20260925 13:14:42.033738 13466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34041
I20260925 13:14:42.033838 13466 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.034027 13466 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
W20260925 13:14:42.034761 13466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34041 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 13:14:42.034842 13466 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.034902 13466 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:42.035277 13466 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.035368 13466 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
W20260925 13:14:42.035720 13466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:14:42.139253 13469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:42.139518 13469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:42.139561 13469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:42.141018 13469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:42.141157 13469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:42.141256 13469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:14:42.143535 13469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41831
--tserver_master_addrs=127.0.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:42.144001 13469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:42.144346 13469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:42.150548 13475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:42.150532 13474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:42.151103 13469 server_base.cc:1034] running on GCE node
W20260925 13:14:42.153816 13477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:42.154107 13469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:42.154492 13469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:42.155833 13469 hybrid_clock.cc:648] HybridClock initialized: now 1790342082155805 us; error 42 us; skew 500 ppm
Sep 25 13:14:42 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:42.158965 13469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:14:42.159520 13469 webserver.cc:460] Webserver started at http://127.0.98.3:41831/ using document root <none> and password file <none>
I20260925 13:14:42.159780 13469 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:42.159885 13469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:14:42 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:42.186223 13469 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 13:14:42.187165 13486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:42.187384 13469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:42.187470 13469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "5f9d2b286d22467486b231222a4e72b1"
format_stamp: "Formatted at 2026-09-25 13:14:41 on dist-test-slave-2d0s"
server_key: "be9f19f8174f038a110d39f0b6277210"
server_key_iv: "076b56b34a1f9bf7c7dd5e23a85504c8"
server_key_version: "kuduclusterkey@0"
I20260925 13:14:42.187937 13469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:42.214406 13469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:42.216971 13469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:42.217165 13469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:42.217399 13469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:42.217785 13469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:42.217839 13469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:42.217875 13469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:42.217892 13469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:42.225086 13469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41727
I20260925 13:14:42.225574 13469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:14:42.226405 13599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41727 every 8 connection(s)
I20260925 13:14:42.229501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13469
Sep 25 13:14:42 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:42.241520 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:42.229649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52897 (local address 127.0.98.62:34041)
0925 13:14:42.229724 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:42.229727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:42.230206 (+ 479us) server_negotiation.cc:407] Connection header received
0925 13:14:42.230874 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:42.230875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:42.230885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:42.230949 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:42.231697 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.232459 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.234082 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.234375 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.234454 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:42.237656 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:42.237657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:42.237657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:42.237692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:42.239715 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:42.240496 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:42.240497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:42.240497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:42.240561 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:42.240850 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:42.240851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:42.240852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:42.240985 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:42.241060 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:42.241332 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 13:14:42.241423 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:14:42.241597 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:42.229663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34041 (local address 127.0.98.3:52897)
0925 13:14:42.229875 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:42.229934 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:42.230755 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:42.231024 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:42.231029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:42.231297 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:42.231585 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:42.231592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.232577 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.232578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:42.233986 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:42.233989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.234541 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.234542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:42.234698 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:42.235791 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:42.235802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:42.237552 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:42.239809 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:42.239816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:42.239820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:42.240163 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:42.240643 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:42.240644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:42.240644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:42.240764 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:42.241059 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:42.241064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:42.241194 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 13:14:42.241278 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:14:42.241399 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260925 13:14:42.242082 13600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34041
I20260925 13:14:42.242193 13600 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.242411 13600 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
W20260925 13:14:42.242956 13600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34041 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 13:14:42.243006 13600 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.243046 13600 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:42.243304 13600 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.243347 13600 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
W20260925 13:14:42.243558 13600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:14:42.245950 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:42.229977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.0.98.62:34041)
0925 13:14:42.230117 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:42.230119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:42.230252 (+ 133us) server_negotiation.cc:407] Connection header received
0925 13:14:42.230349 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:42.230350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:42.230365 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:42.230444 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:42.232575 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.233286 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.235115 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:42.235426 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:42.235499 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:42.237013 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:42.237015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:42.237015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:42.237022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:42.239279 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:42.240264 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:42.240265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:42.240265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:42.240335 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:42.241709 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:42.241710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:42.241711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:42.241845 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:42.242010 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:42.245753 (+ 3743us) server_negotiation.cc:299] Negotiation successful
0925 13:14:42.245848 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
Sep 25 13:14:42 dist-test-slave-2d0s krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:42.832305 13332 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:42.832486 13332 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:42.832948 13133 ts_manager.cc:194] Registered new tserver with Master: 854548e4771b4c5c9c52e74f73eab2aa (127.0.98.1:33345)
I20260925 13:14:42.834007 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44383
I20260925 13:14:43.035902 13466 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:43.036077 13466 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:43.036505 13133 ts_manager.cc:194] Registered new tserver with Master: d63c2a70ca204e2a97950e4d36c1ab7d (127.0.98.2:38197)
I20260925 13:14:43.036949 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51941
Sep 25 13:14:43 dist-test-slave-2d0s krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:14:43.243708 13600 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:43.243888 13600 heartbeater.cc:511] Master 127.0.98.62:34041 requested a full tablet report, sending...
I20260925 13:14:43.244206 13133 ts_manager.cc:194] Registered new tserver with Master: 5f9d2b286d22467486b231222a4e72b1 (127.0.98.3:41727)
I20260925 13:14:43.244619 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52897
I20260925 13:14:43.249009 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:14:43.256505 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.250026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.0.98.62:34041)
0925 13:14:43.250292 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.250302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.250307 (+ 5us) server_negotiation.cc:407] Connection header received
0925 13:14:43.250325 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.250325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.250334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.250411 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:43.250965 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.251476 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.252202 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.252425 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.252500 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.253767 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:43.253768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:43.253769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:43.253775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:43.255357 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:43.255695 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:43.255696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:43.255696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:43.255757 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:43.255954 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:43.255955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:43.255955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:43.256060 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:43.256166 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.256301 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.256378 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":67,"threads_started":1}
I20260925 13:14:43.258486 13133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:14:43.260228 13133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:14:43.269119 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.263402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.0.98.3:41727)
0925 13:14:43.263610 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.263615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.264750 (+ 1135us) server_negotiation.cc:407] Connection header received
0925 13:14:43.265087 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.265090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.265107 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.265195 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:43.265424 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.266523 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.267825 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.268563 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.268622 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.268675 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.268986 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.269035 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":43,"threads_started":1}
I20260925 13:14:43.269550 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.263143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.0.1:58476)
0925 13:14:43.264697 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.264704 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:43.264973 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:43.265293 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:43.265294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:43.265296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:43.265340 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.265342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.266630 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.266631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.267717 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.267721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.268757 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.268758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.268847 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.268863 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 13:14:43.269402 (+ 539us) client_negotiation.cc:240] Negotiation successful
0925 13:14:43.269471 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260925 13:14:43.270121 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.263143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33345 (local address 127.0.0.1:40886)
0925 13:14:43.264494 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.264534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:43.264607 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:43.265060 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:43.265063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:43.265065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:43.265131 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.265133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.266300 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.266301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.267503 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.267506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.269904 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.269905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.269964 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.269967 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:14:43.270018 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:14:43.270053 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"thread_start_us":79,"threads_started":1}
I20260925 13:14:43.270401 13530 tablet_service.cc:1467] Processing CreateTablet for tablet 688b623244054e23975962ae76f32da8 (DEFAULT_TABLE table=test-table [id=9ef859a22ca24e98bd15831a8803dac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:43.270777 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688b623244054e23975962ae76f32da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:43.272444 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.263543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.0.98.1:33345)
0925 13:14:43.263665 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.263668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.264783 (+ 1115us) server_negotiation.cc:407] Connection header received
0925 13:14:43.264821 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.264823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.264838 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.264900 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:43.265197 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.266213 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.267596 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.268259 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.268328 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.268381 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.272334 (+ 3953us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.272375 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260925 13:14:43.272883 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.263356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38197 (local address 127.0.0.1:44694)
0925 13:14:43.268431 (+ 5075us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.268441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:43.268482 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:43.268724 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:43.268725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:43.268726 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:43.268760 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.268762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.270323 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.270324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.271338 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.271341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.272680 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.272681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.272738 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.272742 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:14:43.272791 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:14:43.272825 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5059,"thread_start_us":64,"threads_started":1}
I20260925 13:14:43.273123 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.263466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.0.98.2:38197)
0925 13:14:43.264425 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.264429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.268538 (+ 4109us) server_negotiation.cc:407] Connection header received
0925 13:14:43.268574 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.268577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.268592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.268653 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:43.268824 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.269817 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.271428 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.272077 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.272140 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.272205 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.273018 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.273057 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":66,"threads_started":1}
I20260925 13:14:43.273718 13400 tablet_service.cc:1467] Processing CreateTablet for tablet 688b623244054e23975962ae76f32da8 (DEFAULT_TABLE table=test-table [id=9ef859a22ca24e98bd15831a8803dac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:43.274044 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688b623244054e23975962ae76f32da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:43.274108 13255 tablet_service.cc:1467] Processing CreateTablet for tablet 688b623244054e23975962ae76f32da8 (DEFAULT_TABLE table=test-table [id=9ef859a22ca24e98bd15831a8803dac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:43.274391 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 688b623244054e23975962ae76f32da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:43.277941 13634 tablet_bootstrap.cc:492] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: Bootstrap starting.
I20260925 13:14:43.278568 13634 tablet_bootstrap.cc:654] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:43.278895 13634 log.cc:824] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:43.278901 13633 tablet_bootstrap.cc:492] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d: Bootstrap starting.
I20260925 13:14:43.279633 13634 tablet_bootstrap.cc:492] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: No bootstrap required, opened a new log
I20260925 13:14:43.279688 13634 ts_tablet_manager.cc:1397] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:43.279906 13633 tablet_bootstrap.cc:654] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:43.280182 13633 log.cc:824] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:43.280462 13635 tablet_bootstrap.cc:492] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Bootstrap starting.
I20260925 13:14:43.281041 13635 tablet_bootstrap.cc:654] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:43.281297 13635 log.cc:824] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:43.282034 13635 tablet_bootstrap.cc:492] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: No bootstrap required, opened a new log
I20260925 13:14:43.282095 13635 ts_tablet_manager.cc:1397] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:43.282069 13634 raft_consensus.cc:348] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.282228 13634 raft_consensus.cc:374] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:43.282258 13634 raft_consensus.cc:729] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854548e4771b4c5c9c52e74f73eab2aa, State: Initialized, Role: FOLLOWER
I20260925 13:14:43.282357 13634 consensus_queue.cc:260] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.282563 13634 ts_tablet_manager.cc:1428] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:14:43.282881 13332 heartbeater.cc:503] Master 127.0.98.62:34041 was elected leader, sending a full tablet report...
I20260925 13:14:43.284535 13633 tablet_bootstrap.cc:492] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d: No bootstrap required, opened a new log
I20260925 13:14:43.284477 13635 raft_consensus.cc:348] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.284596 13633 ts_tablet_manager.cc:1397] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 13:14:43.284621 13635 raft_consensus.cc:374] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:43.284655 13635 raft_consensus.cc:729] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f9d2b286d22467486b231222a4e72b1, State: Initialized, Role: FOLLOWER
I20260925 13:14:43.284768 13635 consensus_queue.cc:260] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.285007 13635 ts_tablet_manager.cc:1428] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:14:43.285110 13600 heartbeater.cc:503] Master 127.0.98.62:34041 was elected leader, sending a full tablet report...
I20260925 13:14:43.286895 13633 raft_consensus.cc:348] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.287039 13633 raft_consensus.cc:374] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:43.287066 13633 raft_consensus.cc:729] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63c2a70ca204e2a97950e4d36c1ab7d, State: Initialized, Role: FOLLOWER
I20260925 13:14:43.287158 13633 consensus_queue.cc:260] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.287371 13633 ts_tablet_manager.cc:1428] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:14:43.287416 13466 heartbeater.cc:503] Master 127.0.98.62:34041 was elected leader, sending a full tablet report...
I20260925 13:14:43.312013 13641 raft_consensus.cc:482] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:43.312163 13641 raft_consensus.cc:504] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.312597 13641 leader_election.cc:288] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f9d2b286d22467486b231222a4e72b1 (127.0.98.3:41727), 854548e4771b4c5c9c52e74f73eab2aa (127.0.98.1:33345)
W20260925 13:14:43.316202 13333 tablet.cc:2369] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:14:43.316576 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.313036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39765 (local address 127.0.98.1:33345)
0925 13:14:43.313077 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.313079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.313189 (+ 110us) server_negotiation.cc:407] Connection header received
0925 13:14:43.313284 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.313285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.313296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.313324 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:43.313562 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.314264 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.315645 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.316202 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.316248 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.316257 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.316461 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.316512 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:14:43.316759 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.312897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51083 (local address 127.0.98.3:41727)
0925 13:14:43.312939 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.312941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.313074 (+ 133us) server_negotiation.cc:407] Connection header received
0925 13:14:43.313164 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.313165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.313176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.313202 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:43.313423 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.314147 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.315498 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.316040 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.316081 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.316091 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.316663 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.316702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:14:43.316962 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.312878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33345 (local address 127.0.98.2:39765)
0925 13:14:43.313127 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.313137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:43.313214 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:43.313390 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:43.313391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:43.313392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:43.313463 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.313464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.314382 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.314383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.315545 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.315548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.316315 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.316316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.316382 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.316385 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:14:43.316830 (+ 445us) client_negotiation.cc:240] Negotiation successful
0925 13:14:43.316872 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":69,"threads_started":1}
I20260925 13:14:43.317102 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.312739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.98.2:51083)
0925 13:14:43.313019 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.313029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:43.313098 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:43.313263 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:43.313264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:43.313266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:43.313356 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.313358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.314235 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.314235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.315400 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:43.315403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.316518 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.316519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:43.316590 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.316593 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:14:43.316995 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 13:14:43.317035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
I20260925 13:14:43.317502 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688b623244054e23975962ae76f32da8" candidate_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854548e4771b4c5c9c52e74f73eab2aa" is_pre_election: true
I20260925 13:14:43.317644 13272 raft_consensus.cc:2454] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63c2a70ca204e2a97950e4d36c1ab7d in term 0.
I20260925 13:14:43.317636 13535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688b623244054e23975962ae76f32da8" candidate_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f9d2b286d22467486b231222a4e72b1" is_pre_election: true
I20260925 13:14:43.317816 13535 raft_consensus.cc:2454] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63c2a70ca204e2a97950e4d36c1ab7d in term 0.
I20260925 13:14:43.317873 13356 leader_election.cc:302] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854548e4771b4c5c9c52e74f73eab2aa, d63c2a70ca204e2a97950e4d36c1ab7d; no voters:
I20260925 13:14:43.318092 13641 raft_consensus.cc:2781] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:14:43.318157 13641 raft_consensus.cc:482] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:14:43.318187 13641 raft_consensus.cc:3037] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:43.319047 13641 raft_consensus.cc:504] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.319167 13641 leader_election.cc:288] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [CANDIDATE]: Term 1 election: Requested vote from peers 5f9d2b286d22467486b231222a4e72b1 (127.0.98.3:41727), 854548e4771b4c5c9c52e74f73eab2aa (127.0.98.1:33345)
I20260925 13:14:43.319310 13535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688b623244054e23975962ae76f32da8" candidate_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f9d2b286d22467486b231222a4e72b1"
I20260925 13:14:43.319357 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688b623244054e23975962ae76f32da8" candidate_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854548e4771b4c5c9c52e74f73eab2aa"
I20260925 13:14:43.319379 13535 raft_consensus.cc:3037] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:43.319419 13272 raft_consensus.cc:3037] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:43.320160 13535 raft_consensus.cc:2454] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63c2a70ca204e2a97950e4d36c1ab7d in term 1.
I20260925 13:14:43.320256 13272 raft_consensus.cc:2454] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63c2a70ca204e2a97950e4d36c1ab7d in term 1.
I20260925 13:14:43.320331 13353 leader_election.cc:302] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9d2b286d22467486b231222a4e72b1, d63c2a70ca204e2a97950e4d36c1ab7d; no voters:
I20260925 13:14:43.320526 13641 raft_consensus.cc:2781] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:43.320658 13641 raft_consensus.cc:686] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [term 1 LEADER]: Becoming Leader. State: Replica: d63c2a70ca204e2a97950e4d36c1ab7d, State: Running, Role: LEADER
I20260925 13:14:43.320739 13641 consensus_queue.cc:237] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } }
I20260925 13:14:43.321405 13133 catalog_manager.cc:5495] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d reported cstate change: term changed from 0 to 1, leader changed from <none> to d63c2a70ca204e2a97950e4d36c1ab7d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d63c2a70ca204e2a97950e4d36c1ab7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38197 } health_report { overall_health: HEALTHY } } }
I20260925 13:14:43.328099 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.325330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.0.98.2:38197)
0925 13:14:43.325364 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:43.325366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:43.325372 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:14:43.325398 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:43.325398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:43.325415 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:43.325473 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:14:43.325609 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.326246 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.327221 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:43.327466 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:43.327514 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:43.327666 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:14:43.327764 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:14:43.327836 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:43.328027 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 13:14:43.328043 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:14:43.329710 13272 raft_consensus.cc:1261] T 688b623244054e23975962ae76f32da8 P 854548e4771b4c5c9c52e74f73eab2aa [term 1 FOLLOWER]: Refusing update from remote peer d63c2a70ca204e2a97950e4d36c1ab7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:43.329800 13535 raft_consensus.cc:1261] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 1 FOLLOWER]: Refusing update from remote peer d63c2a70ca204e2a97950e4d36c1ab7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:43.330070 13641 consensus_queue.cc:1035] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "854548e4771b4c5c9c52e74f73eab2aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:43.330196 13644 consensus_queue.cc:1035] T 688b623244054e23975962ae76f32da8 P d63c2a70ca204e2a97950e4d36c1ab7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9d2b286d22467486b231222a4e72b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:43.333196 13647 mvcc.cc:205] Tried to move back new op lower bound from 7333241173316161536 to 7333241173282095104. Current Snapshot: MvccSnapshot[applied={T|T < 7333241173316161536}]
I20260925 13:14:43.333765 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333241173316161536 to 7333241173282095104. Current Snapshot: MvccSnapshot[applied={T|T < 7333241173316161536}]
I20260925 13:14:43.334220 13649 mvcc.cc:205] Tried to move back new op lower bound from 7333241173316161536 to 7333241173282095104. Current Snapshot: MvccSnapshot[applied={T|T < 7333241173316161536}]
I20260925 13:14:43.336027 13132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:14:43.336126 13132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:14:43.336901 13132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f09672174a07441e8341d71c8cb688db: Sending DeleteTablet for 3 replicas of tablet 688b623244054e23975962ae76f32da8
I20260925 13:14:43.337220 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet 688b623244054e23975962ae76f32da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58476
I20260925 13:14:43.337291 13255 tablet_service.cc:1514] Processing DeleteTablet for tablet 688b623244054e23975962ae76f32da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40886
I20260925 13:14:43.337730 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet 688b623244054e23975962ae76f32da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44694
I20260925 13:14:43.338102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13171
W20260925 13:14:43.342391 13120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33345 (error 108)
I20260925 13:14:43.343427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13336
I20260925 13:14:43.344167 13656 tablet_replica.cc:330] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: stopping tablet replica
I20260925 13:14:43.344256 13656 raft_consensus.cc:2229] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:14:43.344341 13656 raft_consensus.cc:2258] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:14:43.344779 13656 ts_tablet_manager.cc:1905] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:14:43.346529 13656 ts_tablet_manager.cc:1918] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:14:43.346586 13656 log.cc:1196] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/688b623244054e23975962ae76f32da8
I20260925 13:14:43.346827 13656 ts_tablet_manager.cc:1939] T 688b623244054e23975962ae76f32da8 P 5f9d2b286d22467486b231222a4e72b1: Deleting consensus metadata
I20260925 13:14:43.347127 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.347025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33345 (local address 127.0.0.1:40900)
0925 13:14:43.347067 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.347100 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:14:43.347203 13117 catalog_manager.cc:4841] TS 5f9d2b286d22467486b231222a4e72b1 (127.0.98.3:41727): tablet 688b623244054e23975962ae76f32da8 (table test-table [id=9ef859a22ca24e98bd15831a8803dac1]) successfully deleted
W20260925 13:14:43.347290 13120 catalog_manager.cc:4584] TS 854548e4771b4c5c9c52e74f73eab2aa (127.0.98.1:33345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 688b623244054e23975962ae76f32da8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33345: connect: Connection refused (error 111)
I20260925 13:14:43.351029 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:43.350936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38197 (local address 127.0.0.1:44698)
0925 13:14:43.350979 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:43.351003 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 13:14:43.351138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13469
I20260925 13:14:43.356391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13100
I20260925 13:14:43.361339 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 13:14:43.706036 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 13:14:43.706199 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 13:14:44.050767 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 13:14:44.050 UTC [12167] LOG: received smart shutdown request
2026-09-25 13:14:44.054 UTC [12167] LOG: background worker "logical replication launcher" (PID 12176) exited with exit code 1
2026-09-25 13:14:44.054 UTC [12171] LOG: shutting down
2026-09-25 13:14:44.218 UTC [12167] LOG: database system is shut down
2026-09-25T13:14:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48196 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:14:44 dist-test-slave-2d0s krb5kdc[13668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:14:44 dist-test-slave-2d0s krb5kdc[13668](info): set up 2 sockets
Sep 25 13:14:44 dist-test-slave-2d0s krb5kdc[13668](info): commencing operation
krb5kdc: starting...
W20260925 13:14:45.353561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:14:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:14:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:14:45.472086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46335,127.0.98.61:38263,127.0.98.60:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:45.551662 13681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:45.551867 13681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:45.551888 13681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:45.553041 13681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:45.553083 13681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:45.553098 13681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:45.553112 13681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:45.553123 13681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:45.554672 13681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46335,127.0.98.61:38263,127.0.98.60:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:45.554924 13681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:45.555171 13681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:45.560040 13689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:45.560073 13687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:45.560078 13681 server_base.cc:1034] running on GCE node
W20260925 13:14:45.560045 13686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:45.560565 13681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:45.560849 13681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:45.562011 13681 hybrid_clock.cc:648] HybridClock initialized: now 1790342085561997 us; error 29 us; skew 500 ppm
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:45.571233 13681 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:45.571709 13681 webserver.cc:460] Webserver started at http://127.0.98.62:43963/ using document root <none> and password file <none>
I20260925 13:14:45.571952 13681 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:45.572026 13681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:45.572181 13681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:45.573294 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "704934133ed343518702394afdfb386c"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "6725c2c24214d43922b390c9e1a1f21d"
server_key_iv: "dc27acd02e689ab00b1cfea5a906aff7"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.573719 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "704934133ed343518702394afdfb386c"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "6725c2c24214d43922b390c9e1a1f21d"
server_key_iv: "dc27acd02e689ab00b1cfea5a906aff7"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.575107 13681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:14:45.575918 13696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:45.576130 13681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:14:45.576246 13681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "704934133ed343518702394afdfb386c"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "6725c2c24214d43922b390c9e1a1f21d"
server_key_iv: "dc27acd02e689ab00b1cfea5a906aff7"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.576329 13681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:45.599165 13681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:45.601739 13681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:45.601918 13681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:45.605793 13681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46335
I20260925 13:14:45.605851 13748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46335 every 8 connection(s)
I20260925 13:14:45.606222 13681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:45.607075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13681
I20260925 13:14:45.607153 13749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:45.607195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:14:45.607409 392 external_mini_cluster.cc:1428] Setting key 4d0fe8e8683efe130899bae3cb8bd837
I20260925 13:14:45.609421 13749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:45.622087 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.610184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46335 (local address 127.0.0.1:33504)
0925 13:14:45.610399 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.610439 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.610946 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.611241 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.611244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.611521 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.611674 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.611678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.612701 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.612702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.613379 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.613380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.613947 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.613948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.614017 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.614569 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.614577 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.618207 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.620496 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.620503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.620505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.620717 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.621017 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.621018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.621018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.621124 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.621587 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.621592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.621649 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.621775 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.621918 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260925 13:14:45.622200 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.610211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.0.98.62:46335)
0925 13:14:45.610409 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.610418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.610547 (+ 129us) server_negotiation.cc:407] Connection header received
0925 13:14:45.611111 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.611113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.611129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.611223 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.611776 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.612593 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.613471 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.613843 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.613970 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.618306 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.618313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.618313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.618320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.620384 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.620805 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.620808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.620809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.620918 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.621197 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.621198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.621198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.621513 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.621607 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.621837 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.622021 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260925 13:14:45.622886 13749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:14:45.623224 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.623121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33136)
0925 13:14:45.623167 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.623206 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:14:45.623381 13697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
I20260925 13:14:45.623755 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.623695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33148)
0925 13:14:45.623719 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.623741 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 13:14:45.623870 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38263: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 13:14:45.632442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38263
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46335,127.0.98.61:38263,127.0.98.60:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:45.673105 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } attempt: 1
I20260925 13:14:45.673789 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.673514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33154)
0925 13:14:45.673584 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.673708 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:14:45.674304 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.674185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33164)
0925 13:14:45.674243 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.674277 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 13:14:45.674413 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38263: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
W20260925 13:14:45.713598 13756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:45.713896 13756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:45.713943 13756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:45.715090 13756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:45.715158 13756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:45.715184 13756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:45.715214 13756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:45.715240 13756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:45.716729 13756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46335,127.0.98.61:38263,127.0.98.60:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:45.716995 13756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:45.717229 13756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:45.722064 13764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:45.722040 13761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:45.722182 13756 server_base.cc:1034] running on GCE node
W20260925 13:14:45.722082 13762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:45.722456 13756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:45.722765 13756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:45.723937 13756 hybrid_clock.cc:648] HybridClock initialized: now 1790342085723916 us; error 39 us; skew 500 ppm
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:45.726511 13756 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 13:14:45.726987 13756 webserver.cc:460] Webserver started at http://127.0.98.61:42331/ using document root <none> and password file <none>
I20260925 13:14:45.727221 13756 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:45.727265 13756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:45.727396 13756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:45.728441 13756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data/instance:
uuid: "1177b857faa24337a4038c9318a15d41"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "6a6d916f5aee4505ba3e052468eb089d"
server_key_iv: "e7d97283ccafe0d9961ecec12935d5c1"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.728791 13756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal/instance:
uuid: "1177b857faa24337a4038c9318a15d41"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "6a6d916f5aee4505ba3e052468eb089d"
server_key_iv: "e7d97283ccafe0d9961ecec12935d5c1"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.730159 13756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 13:14:45.730873 13771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:45.731034 13756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:45.731134 13756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal
uuid: "1177b857faa24337a4038c9318a15d41"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "6a6d916f5aee4505ba3e052468eb089d"
server_key_iv: "e7d97283ccafe0d9961ecec12935d5c1"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.731204 13756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:45.740247 13756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:45.742609 13756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:45.742569 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } attempt: 2
I20260925 13:14:45.742789 13756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:45.743073 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.742879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33174)
0925 13:14:45.742971 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.743019 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 13:14:45.743476 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.743353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33178)
0925 13:14:45.743406 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.743446 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 13:14:45.743610 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38263: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38263: connect: Connection refused (error 111)
I20260925 13:14:45.746775 13756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38263
I20260925 13:14:45.746803 13823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38263 every 8 connection(s)
I20260925 13:14:45.747182 13756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:14:45.748039 13824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:45.749557 13824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:45.757332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13756
I20260925 13:14:45.757457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-1/wal/instance
I20260925 13:14:45.757704 392 external_mini_cluster.cc:1428] Setting key 4047bb4570c46f2f90142f0e42c122b7
I20260925 13:14:45.758606 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.750490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.0.98.62:46335)
0925 13:14:45.750547 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.750548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.750778 (+ 230us) server_negotiation.cc:407] Connection header received
0925 13:14:45.751281 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.751283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.751294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.751344 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.751965 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.752466 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.753527 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.753797 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.753882 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.755878 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.755880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.755880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.755888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.757143 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.757596 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.757597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.757597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.757693 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.757954 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.757955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.757956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.758077 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.758150 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.758430 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.758517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:14:45.758803 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.750463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46335 (local address 127.0.0.1:33516)
0925 13:14:45.750658 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.750693 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.751169 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.751417 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.751420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.751669 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.751860 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.751864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.752614 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.752615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.753427 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.753428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.753890 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.753891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.754017 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.754489 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.754497 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.755780 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.757238 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.757244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.757246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.757533 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.757747 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.757748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.757748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.757875 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.758150 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.758158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.758296 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.758398 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.758564 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260925 13:14:45.759541 13824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 13:14:45.767768 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.759812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33180)
0925 13:14:45.759862 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.760015 (+ 153us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.760050 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.760235 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.760236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.760358 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.760406 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.760407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.761730 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.761731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.762237 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.762239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.762932 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.762933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.762984 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.763400 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.763412 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.764758 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.766588 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.766589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.766589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.766769 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.766970 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.766971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.766971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.767023 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.767510 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.767511 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.767558 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.767628 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.767699 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:14:45.767890 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.759905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.0.98.61:38263)
0925 13:14:45.760057 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.760062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.760096 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:14:45.760128 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.760130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.760145 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.760219 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.760476 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.761627 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.762322 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.762846 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.762924 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.764847 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.764855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.764855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.764863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.766457 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.766840 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.766842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.766843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.766894 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.767112 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.767113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.767113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.767273 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.767417 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.767700 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.767781 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 13:14:45.768518 13824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:14:45.768911 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.768777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60778)
0925 13:14:45.768844 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.768893 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 13:14:45.769069 13772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260925 13:14:45.769492 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.769403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60790)
0925 13:14:45.769450 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.769476 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 13:14:45.769616 13824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 13:14:45.785058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40541
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46335,127.0.98.61:38263,127.0.98.60:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:45.818892 13824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 1
I20260925 13:14:45.819406 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.819260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60794)
0925 13:14:45.819334 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.819376 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 13:14:45.819818 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.819730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60796)
0925 13:14:45.819776 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.819801 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 13:14:45.819947 13824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260925 13:14:45.834772 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } attempt: 3
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:14:45.842626 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.835141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33184)
0925 13:14:45.835214 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.835231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.835310 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.835475 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.835476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.835653 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.835725 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.835726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.836721 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.836722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.837253 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.837254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.837763 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.837764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.837827 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.838355 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.838360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.839837 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.841602 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.841603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.841604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.841798 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.842005 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.842005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.842006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.842060 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.842334 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.842335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.842380 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.842446 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.842540 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:14:45.842665 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.835237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.0.98.61:38263)
0925 13:14:45.835313 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.835315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.835358 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:14:45.835387 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.835388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.835402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.835454 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.835805 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.836601 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.837351 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.837690 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.837755 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.839928 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.839930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.839931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.839939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.841509 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.841866 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.841867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.841868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.841937 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.842119 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.842120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.842121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.842250 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.842316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.842484 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.842575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 13:14:45.843261 13749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:14:45.843627 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.843512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60800)
0925 13:14:45.843566 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.843601 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 13:14:45.843958 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.843863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60810)
0925 13:14:45.843912 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.843938 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:14:45.844084 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
W20260925 13:14:45.870263 13831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:45.870474 13831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:45.870497 13831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:45.871608 13831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:45.871644 13831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:45.871658 13831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:45.871670 13831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:45.871682 13831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:45.873123 13831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46335,127.0.98.61:38263,127.0.98.60:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:45.873317 13831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:45.873492 13831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:14:45.875233 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 1
I20260925 13:14:45.875698 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.875547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60820)
0925 13:14:45.875614 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.875661 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:14:45.876102 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.876005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60824)
0925 13:14:45.876051 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.876082 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:14:45.876269 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
W20260925 13:14:45.878741 13836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:45.878758 13837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:45.878757 13831 server_base.cc:1034] running on GCE node
W20260925 13:14:45.878741 13839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:45.879181 13831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:45.879495 13831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:45.880662 13831 hybrid_clock.cc:648] HybridClock initialized: now 1790342085880651 us; error 41 us; skew 500 ppm
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:45.883124 13831 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 13:14:45.883589 13831 webserver.cc:460] Webserver started at http://127.0.98.60:41387/ using document root <none> and password file <none>
I20260925 13:14:45.883872 13831 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:45.883945 13831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:45.884092 13831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:45.885378 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data/instance:
uuid: "4fed8a568144438eab7e58f67f036d09"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "76ea1ee80a873daa2ec896dfe3b53c4f"
server_key_iv: "8364ae5354a52ec48529a67467581074"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.885761 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal/instance:
uuid: "4fed8a568144438eab7e58f67f036d09"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "76ea1ee80a873daa2ec896dfe3b53c4f"
server_key_iv: "8364ae5354a52ec48529a67467581074"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.887034 13831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:45.887701 13846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:45.887977 13831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:45.888091 13831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal
uuid: "4fed8a568144438eab7e58f67f036d09"
format_stamp: "Formatted at 2026-09-25 13:14:45 on dist-test-slave-2d0s"
server_key: "76ea1ee80a873daa2ec896dfe3b53c4f"
server_key_iv: "8364ae5354a52ec48529a67467581074"
server_key_version: "encryptionkey@0"
I20260925 13:14:45.888092 13824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 2
I20260925 13:14:45.888167 13831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:45.888477 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.888355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60826)
0925 13:14:45.888410 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.888441 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:14:45.888789 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.888696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60828)
0925 13:14:45.888742 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.888766 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:14:45.888886 13824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260925 13:14:45.901274 13831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:45.903471 13831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:45.903626 13831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:45.907667 13831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40541
I20260925 13:14:45.907742 13898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40541 every 8 connection(s)
I20260925 13:14:45.908108 13831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:14:45.909066 13899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:45.910722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13831
I20260925 13:14:45.910809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790341932215442-392-0/minicluster-data/master-2/wal/instance
I20260925 13:14:45.910738 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:14:45.911026 392 external_mini_cluster.cc:1428] Setting key 5cc034c220ad178004e2bcf5c99f1665
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:14:45.920360 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.911558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.0.98.62:46335)
0925 13:14:45.911613 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.911615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.911845 (+ 230us) server_negotiation.cc:407] Connection header received
0925 13:14:45.912399 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.912401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.912412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.912484 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.913054 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.913540 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.914489 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.914681 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.914764 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.917186 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.917188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.917188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.917195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.918889 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.919292 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.919293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.919294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.919354 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.919598 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.919599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.919599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.919736 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.919785 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.920182 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.920266 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:14:45.920765 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.911546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46335 (local address 127.0.0.1:33518)
0925 13:14:45.911725 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.911761 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.912280 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.912509 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.912512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.912791 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.912957 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.912962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.913622 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.913624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.914390 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.914393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.914771 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.914772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.914911 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.915472 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.915483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.917094 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.918972 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.918979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.918980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.919215 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.919434 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.919435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.919435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.919532 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.919805 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.919812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.920005 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.920142 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.920296 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260925 13:14:45.920780 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.911749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.0.98.60:40541)
0925 13:14:45.911884 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.911889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.911922 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:14:45.912297 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.912306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.912345 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.912449 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.913103 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.913999 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.914733 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.915026 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.915103 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.917699 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.917708 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.917709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.917716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.919554 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.919940 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.919943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.919943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.919992 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.920195 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.920196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.920197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.920367 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.920567 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.920607 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.920666 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260925 13:14:45.921375 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:45.928740 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.921687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38263 (local address 127.0.0.1:33190)
0925 13:14:45.921728 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.921739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.921776 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.921957 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.921958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.922120 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.922184 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.922185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.923028 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.923029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.923725 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.923727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.924084 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.924084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.924138 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.924558 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.924563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.925812 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.927723 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.927724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.927724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.927905 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.928119 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.928120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.928121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.928179 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.928476 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.928477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.928525 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.928595 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.928678 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 13:14:45.928812 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.921771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.0.98.61:38263)
0925 13:14:45.921816 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.921818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.921826 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:14:45.921852 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.921853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.921863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.921940 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.922260 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.922949 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.923800 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.924003 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.924069 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.925915 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.925916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.925916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.925924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.927640 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.927990 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.927991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.927992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.928051 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.928246 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.928247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.928247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.928390 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.928463 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.928626 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.928703 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:14:45.929359 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.921751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.98.62:46335)
0925 13:14:45.921794 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.921797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.921841 (+ 44us) server_negotiation.cc:407] Connection header received
0925 13:14:45.921911 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.921911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.921921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.921969 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.922504 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.923181 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.924021 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.924247 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.924320 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.926484 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.926485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.926486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.926493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.927979 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.928357 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.928358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.928358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.928408 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.928607 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.928608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.928608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.928726 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.929156 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.929199 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.929279 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:14:45.929436 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:14:45.936712 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.929727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.60:40541)
0925 13:14:45.929809 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.929811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.929819 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:14:45.929838 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.929838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.929850 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.929913 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.930195 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.930898 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.931674 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.932004 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.932077 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.933987 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.933988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.933989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.933996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.935488 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.935863 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.935864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.935865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.935923 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.936158 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.936159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.936159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.936276 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.936343 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.936506 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.936624 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":63}
I20260925 13:14:45.936718 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.929697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60834)
0925 13:14:45.929727 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.929737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.929773 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.929919 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.929920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.930086 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.930123 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.930125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.930970 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.930971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.931605 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.931606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.932126 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.932127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.932187 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.932634 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.932638 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.933903 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.935582 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.935583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.935584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.935795 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.935992 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.935993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.935993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.936094 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.936357 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.936358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.936401 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.936481 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.936582 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 13:14:45.937922 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.930030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.0.98.61:38263)
0925 13:14:45.930071 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.930073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.930080 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:14:45.930099 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.930099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.930108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.930175 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.930717 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.931331 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.932142 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.932432 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.932522 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.935040 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.935041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.935041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.935048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.936772 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.937101 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.937103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.937103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.937155 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.937346 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.937347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.937347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.937461 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.937600 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.937704 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.937802 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:14:45.938546 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09: Bootstrap starting.
I20260925 13:14:45.939621 13899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:45.939653 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:14:45.939942 13899 log.cc:824] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:45.940789 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09: No bootstrap required, opened a new log
I20260925 13:14:45.942876 13899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:45.943053 13899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:45.943099 13899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fed8a568144438eab7e58f67f036d09, State: Initialized, Role: FOLLOWER
I20260925 13:14:45.943226 13899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:45.943496 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:45.943542 13899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:45.945078 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:14:45.948037 13923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:14:45.948092 13923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:14:45.946110 13831 master_runner.cc:186] Error getting master registration for 127.0.98.62:46335: OK, instance_id { permanent_uuid: "704934133ed343518702394afdfb386c" instance_seqno: 1790342085604676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:14:45.949555 13831 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:14:45.949966 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.941235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.0.98.61:38263)
0925 13:14:45.941279 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.941281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.941413 (+ 132us) server_negotiation.cc:407] Connection header received
0925 13:14:45.941506 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.941508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.941518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.941546 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.942277 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.942950 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.943986 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.944304 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.944356 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.946602 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.946603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.946603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.946610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.948702 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.949101 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.949102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.949102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.949155 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.949343 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.949344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.949344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.949466 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.949551 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.949816 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.949903 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:14:45.950459 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.941196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.0.98.62:46335)
0925 13:14:45.941280 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.941282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.941442 (+ 160us) server_negotiation.cc:407] Connection header received
0925 13:14:45.941531 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.941532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.941543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.941588 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.942135 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.942893 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.943998 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.944334 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.944393 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.946063 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.946064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.946064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.946071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.947825 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.948625 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.948626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.948626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.948687 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.948903 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.948904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.948905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.949297 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.949939 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.950240 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.950330 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 13:14:45.951211 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.941418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.0.98.60:40541)
0925 13:14:45.941453 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.941455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.943685 (+ 2230us) server_negotiation.cc:407] Connection header received
0925 13:14:45.943770 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.943771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.943780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.943807 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.944501 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.945187 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.946314 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.946581 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.946627 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.948158 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.948159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.948159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.948165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.949874 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.950292 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.950293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.950293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.950348 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.950534 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.950534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.950535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.950693 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.950797 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.951014 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.951091 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:14:45.951397 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 2
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:14:45.958232 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.951660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.0.98.60:40541)
0925 13:14:45.951778 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.951779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.951785 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:14:45.951799 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.951799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.951806 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.951877 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.952122 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.952584 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.953242 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.953448 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.953524 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.955434 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.955435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.955436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.955443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.956965 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.957363 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.957364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.957365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.957420 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.957640 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.957641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.957641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.957828 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.957895 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.958104 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.958172 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 13:14:45.958237 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.951618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60848)
0925 13:14:45.951665 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.951675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.951708 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.951881 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.951882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.952018 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.952050 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.952051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.952665 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.952666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.953159 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.953160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.953520 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.953521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.953578 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.954038 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.954042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.955349 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.957063 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.957064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.957065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.957268 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.957491 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.957492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.957492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.957551 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.957908 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.957909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.957958 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.958037 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.958135 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:14:45.959658 13749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: Bootstrap starting.
I20260925 13:14:45.960335 13749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:45.960628 13749 log.cc:824] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:45.961809 13749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: No bootstrap required, opened a new log
I20260925 13:14:45.964011 13749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:45.964157 13749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:45.964198 13749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704934133ed343518702394afdfb386c, State: Initialized, Role: FOLLOWER
I20260925 13:14:45.964521 13749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:45.964848 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:45.964890 13749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:45.964952 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:14:45.966737 13937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:14:45.966858 13937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:14:45.967532 13681 master_runner.cc:186] Error getting master registration for 127.0.98.61:38263: OK, instance_id { permanent_uuid: "1177b857faa24337a4038c9318a15d41" instance_seqno: 1790342085745613 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:14:45.970490 13681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:14:45.980026 13824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 3
Sep 25 13:14:45 dist-test-slave-2d0s krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:14:45.988335 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.980358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:60854)
0925 13:14:45.980432 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:45.980445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:45.980497 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:45.980671 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:45.980671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:45.980873 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:45.980927 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.980929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.981898 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.981899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.982724 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:45.982727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.983263 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.983264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:45.983325 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.983816 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:45.983821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:45.985379 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:45.987236 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.987237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.987238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.987459 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.987671 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:45.987672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:45.987672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:45.987739 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:45.988007 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:45.988007 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:45.988061 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 13:14:45.988133 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:14:45.988229 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 13:14:45.988341 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:45.980453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.0.98.60:40541)
0925 13:14:45.980519 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:45.980521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:45.980535 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:14:45.980568 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:45.980570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:45.980583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:45.980659 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:45.981019 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.981810 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.982813 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:45.983137 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:45.983204 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:45.985466 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:45.985468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:45.985468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:45.985476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:45.987147 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.987537 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.987538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.987538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.987599 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:45.987807 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:45.987808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:45.987808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:45.987937 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:45.988003 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:45.988171 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:14:45.988247 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:14:45.990254 13824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: Bootstrap starting.
I20260925 13:14:45.991061 13824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:45.991410 13824 log.cc:824] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:45.992302 13824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: No bootstrap required, opened a new log
I20260925 13:14:45.994571 13824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:45.994714 13824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:45.994748 13824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1177b857faa24337a4038c9318a15d41, State: Initialized, Role: FOLLOWER
I20260925 13:14:45.994894 13824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:45.995223 13824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:45.995186 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:45.995373 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:14:45.996987 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:14:45.997110 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:14:45.999133 13756 master_runner.cc:263] No leader master found from master 1177b857faa24337a4038c9318a15d41
I20260925 13:14:45.999188 13756 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:14:46.033804 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:46.033999 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:46.034292 13912 leader_election.cc:288] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704934133ed343518702394afdfb386c (127.0.98.62:46335), 1177b857faa24337a4038c9318a15d41 (127.0.98.61:38263)
I20260925 13:14:46.034674 13723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fed8a568144438eab7e58f67f036d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704934133ed343518702394afdfb386c" is_pre_election: true
I20260925 13:14:46.034863 13723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fed8a568144438eab7e58f67f036d09 in term 0.
I20260925 13:14:46.035104 13848 leader_election.cc:302] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fed8a568144438eab7e58f67f036d09, 704934133ed343518702394afdfb386c; no voters:
I20260925 13:14:46.035523 13798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fed8a568144438eab7e58f67f036d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1177b857faa24337a4038c9318a15d41" is_pre_election: true
I20260925 13:14:46.035703 13798 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fed8a568144438eab7e58f67f036d09 in term 0.
I20260925 13:14:46.036857 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:14:46.036937 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:14:46.036969 13912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:46.037973 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:46.038113 13912 leader_election.cc:288] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [CANDIDATE]: Term 1 election: Requested vote from peers 704934133ed343518702394afdfb386c (127.0.98.62:46335), 1177b857faa24337a4038c9318a15d41 (127.0.98.61:38263)
I20260925 13:14:46.038316 13723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fed8a568144438eab7e58f67f036d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704934133ed343518702394afdfb386c"
I20260925 13:14:46.038410 13723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:46.039525 13723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fed8a568144438eab7e58f67f036d09 in term 1.
I20260925 13:14:46.039738 13848 leader_election.cc:302] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fed8a568144438eab7e58f67f036d09, 704934133ed343518702394afdfb386c; no voters:
I20260925 13:14:46.039986 13798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fed8a568144438eab7e58f67f036d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1177b857faa24337a4038c9318a15d41"
I20260925 13:14:46.040118 13798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:46.041131 13798 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fed8a568144438eab7e58f67f036d09 in term 1.
I20260925 13:14:46.041481 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:46.041695 13912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [term 1 LEADER]: Becoming Leader. State: Replica: 4fed8a568144438eab7e58f67f036d09, State: Running, Role: LEADER
I20260925 13:14:46.041803 13912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260925 13:14:46.042229 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fed8a568144438eab7e58f67f036d09. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.042332 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:46.043064 13955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:46.043268 13955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:14:46.044950 13798 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [term 1 FOLLOWER]: Refusing update from remote peer 4fed8a568144438eab7e58f67f036d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:46.045367 13953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:46.046743 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fed8a568144438eab7e58f67f036d09. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.046842 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:14:46.047211 13723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [term 1 FOLLOWER]: Refusing update from remote peer 4fed8a568144438eab7e58f67f036d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:46.047499 13912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:46.048678 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fed8a568144438eab7e58f67f036d09. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.048772 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:14:46.050414 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.050504 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:46.050992 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.051096 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:14:46.051297 13956 mvcc.cc:205] Tried to move back new op lower bound from 7333241184437710848 to 7333241184427933696. Current Snapshot: MvccSnapshot[applied={T|T < 7333241184437710848}]
I20260925 13:14:46.051403 13957 mvcc.cc:205] Tried to move back new op lower bound from 7333241184437710848 to 7333241184427933696. Current Snapshot: MvccSnapshot[applied={T|T < 7333241184437710848}]
I20260925 13:14:46.051985 13955 catalog_manager.cc:1347] Generated new cluster ID: 49c546f683ff4f7282b5b4d4862dd065
I20260925 13:14:46.052052 13955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:14:46.059203 13955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:14:46.059921 13955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:14:46.060272 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.060370 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:46.061026 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fed8a568144438eab7e58f67f036d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704934133ed343518702394afdfb386c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46335 } } peers { permanent_uuid: "1177b857faa24337a4038c9318a15d41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38263 } } peers { permanent_uuid: "4fed8a568144438eab7e58f67f036d09" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260925 13:14:46.061129 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:14:46.061196 13959 mvcc.cc:205] Tried to move back new op lower bound from 7333241184437710848 to 7333241184427933696. Current Snapshot: MvccSnapshot[applied={T|T < 7333241184437710848}]
I20260925 13:14:46.078976 13955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fed8a568144438eab7e58f67f036d09: Generated new TSK 0
I20260925 13:14:46.079193 13955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:14:46.967378 13937 catalog_manager.cc:1259] Loaded cluster ID: 49c546f683ff4f7282b5b4d4862dd065
I20260925 13:14:46.967461 13937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: loading cluster ID for follower catalog manager: success
I20260925 13:14:46.968676 13937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: acquiring CA information for follower catalog manager: success
I20260925 13:14:46.968955 13937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 704934133ed343518702394afdfb386c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:14:46.997583 13951 catalog_manager.cc:1259] Loaded cluster ID: 49c546f683ff4f7282b5b4d4862dd065
I20260925 13:14:46.997681 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: loading cluster ID for follower catalog manager: success
I20260925 13:14:46.998517 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: acquiring CA information for follower catalog manager: success
I20260925 13:14:46.998834 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1177b857faa24337a4038c9318a15d41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:14:47.133716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13681
I20260925 13:14:47.138887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13756
I20260925 13:14:47.144328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13831
2026-09-25T13:14:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2818 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:14:47 dist-test-slave-2d0s krb5kdc[13973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:14:47 dist-test-slave-2d0s krb5kdc[13973](info): set up 2 sockets
Sep 25 13:14:47 dist-test-slave-2d0s krb5kdc[13973](info): commencing operation
krb5kdc: starting...
W20260925 13:14:48.173846 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:14:48 dist-test-slave-2d0s krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:14:48.292654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:48.373641 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:48.373999 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:48.374038 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:48.375227 13986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:48.375304 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:48.375329 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:48.375349 13986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:48.375379 13986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:48.376880 13986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:48.377177 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:48.377413 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:48.382299 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:48.382333 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:48.382560 13986 server_base.cc:1034] running on GCE node
W20260925 13:14:48.382296 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:48.382884 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:48.383208 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:48.384387 13986 hybrid_clock.cc:648] HybridClock initialized: now 1790342088384374 us; error 25 us; skew 500 ppm
Sep 25 13:14:48 dist-test-slave-2d0s krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:48.386986 13986 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:48.387380 13986 webserver.cc:460] Webserver started at http://127.0.98.62:45295/ using document root <none> and password file <none>
I20260925 13:14:48.387576 13986 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:48.387616 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:48.387702 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:48.388736 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "23afdc20608847db9c27d65c2b8e912e"
format_stamp: "Formatted at 2026-09-25 13:14:48 on dist-test-slave-2d0s"
server_key: "474c308eea235ec6006ad82a1150bcfc"
server_key_iv: "2d744d1139ab239dc7c39d38c76cb8b3"
server_key_version: "encryptionkey@0"
I20260925 13:14:48.389034 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "23afdc20608847db9c27d65c2b8e912e"
format_stamp: "Formatted at 2026-09-25 13:14:48 on dist-test-slave-2d0s"
server_key: "474c308eea235ec6006ad82a1150bcfc"
server_key_iv: "2d744d1139ab239dc7c39d38c76cb8b3"
server_key_version: "encryptionkey@0"
I20260925 13:14:48.390381 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:14:48.391060 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:48.391252 13986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:48.391340 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "23afdc20608847db9c27d65c2b8e912e"
format_stamp: "Formatted at 2026-09-25 13:14:48 on dist-test-slave-2d0s"
server_key: "474c308eea235ec6006ad82a1150bcfc"
server_key_iv: "2d744d1139ab239dc7c39d38c76cb8b3"
server_key_version: "encryptionkey@0"
I20260925 13:14:48.391396 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:48.402295 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:48.404727 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:48.404841 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:48.408766 13986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45243
I20260925 13:14:48.408856 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45243 every 8 connection(s)
I20260925 13:14:48.409198 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:48.410173 14054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:48.412738 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e: Bootstrap starting.
I20260925 13:14:48.413394 14054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:48.413736 14054 log.cc:824] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:48.414487 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e: No bootstrap required, opened a new log
I20260925 13:14:48.416043 14054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23afdc20608847db9c27d65c2b8e912e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260925 13:14:48.416208 14054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:48.416254 14054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23afdc20608847db9c27d65c2b8e912e, State: Initialized, Role: FOLLOWER
I20260925 13:14:48.416406 14054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23afdc20608847db9c27d65c2b8e912e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260925 13:14:48.416491 14054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:14:48.416533 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:14:48.416582 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:48.417119 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23afdc20608847db9c27d65c2b8e912e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260925 13:14:48.417237 14054 leader_election.cc:302] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23afdc20608847db9c27d65c2b8e912e; no voters:
I20260925 13:14:48.417430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 13986
I20260925 13:14:48.417448 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:14:48.417475 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:48.417550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:14:48.417588 14057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [term 1 LEADER]: Becoming Leader. State: Replica: 23afdc20608847db9c27d65c2b8e912e, State: Running, Role: LEADER
I20260925 13:14:48.417745 14054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:48.417793 392 external_mini_cluster.cc:1428] Setting key 6d661aa4c00974ec2a40f2003b7a96d6
I20260925 13:14:48.417763 14057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23afdc20608847db9c27d65c2b8e912e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260925 13:14:48.418102 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23afdc20608847db9c27d65c2b8e912e. Latest consensus state: current_term: 1 leader_uuid: "23afdc20608847db9c27d65c2b8e912e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23afdc20608847db9c27d65c2b8e912e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } } }
I20260925 13:14:48.418171 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:48.418478 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23afdc20608847db9c27d65c2b8e912e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23afdc20608847db9c27d65c2b8e912e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } } }
I20260925 13:14:48.418547 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:48.419417 14062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:48.419966 14062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:14:48.421437 14062 catalog_manager.cc:1347] Generated new cluster ID: 54a9224a0d8a495caa87bc6b05701525
I20260925 13:14:48.421497 14062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:14:48 dist-test-slave-2d0s krb5kdc[13973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:48.428251 14062 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:14:48.428951 14062 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:14:48.430737 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:48.418406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.0.98.62:45243)
0925 13:14:48.418847 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:48.418857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:48.418888 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:14:48.419022 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:48.419029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:48.419084 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:48.419225 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:48.420129 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:48.421501 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:48.422639 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:48.423279 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:48.423400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:48.426242 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:48.426246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:48.426247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:48.426260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:48.429107 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:48.429501 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:48.429503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:48.429504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:48.429616 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:48.429844 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:48.429845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:48.429845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:48.430128 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:48.430261 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:48.430377 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 13:14:48.430514 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":65,"threads_started":1}
I20260925 13:14:48.431603 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:14:48.441766 14062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23afdc20608847db9c27d65c2b8e912e: Generated new TSK 0
I20260925 13:14:48.441996 14062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:14:48.485594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 13986
2026-09-25T13:14:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1339 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:14:48 dist-test-slave-2d0s krb5kdc[14081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:14:48 dist-test-slave-2d0s krb5kdc[14081](info): set up 2 sockets
Sep 25 13:14:48 dist-test-slave-2d0s krb5kdc[14081](info): commencing operation
krb5kdc: starting...
W20260925 13:14:49.510013 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:14:49 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:14:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:14:49.628600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:49.711488 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:49.711692 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:49.711715 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:49.712816 14094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:49.712852 14094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:49.712867 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:49.712878 14094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:49.712891 14094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:49.714282 14094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:49.714478 14094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:49.714649 14094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:49.719345 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:49.719365 14100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:49.719355 14099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:49.719626 14094 server_base.cc:1034] running on GCE node
I20260925 13:14:49.719853 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:49.720194 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:49.721354 14094 hybrid_clock.cc:648] HybridClock initialized: now 1790342089721332 us; error 31 us; skew 500 ppm
Sep 25 13:14:49 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:49.723847 14094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:49.724295 14094 webserver.cc:460] Webserver started at http://127.0.98.62:37701/ using document root <none> and password file <none>
I20260925 13:14:49.724541 14094 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:49.724612 14094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:49.724757 14094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:49.725904 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "9364433dedec48fda122bb8fb5449f28"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "edd2e4b70d9317f522417c943a579fa1"
server_key_iv: "ad0135cdca8c2cbb75224c266c310dc7"
server_key_version: "encryptionkey@0"
I20260925 13:14:49.726279 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "9364433dedec48fda122bb8fb5449f28"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "edd2e4b70d9317f522417c943a579fa1"
server_key_iv: "ad0135cdca8c2cbb75224c266c310dc7"
server_key_version: "encryptionkey@0"
I20260925 13:14:49.727658 14094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:14:49.728353 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:49.728543 14094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:49.728647 14094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "9364433dedec48fda122bb8fb5449f28"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "edd2e4b70d9317f522417c943a579fa1"
server_key_iv: "ad0135cdca8c2cbb75224c266c310dc7"
server_key_version: "encryptionkey@0"
I20260925 13:14:49.728720 14094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:49.741186 14094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:49.743206 14094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:49.743378 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:49.747510 14094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35321
I20260925 13:14:49.747542 14161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35321 every 8 connection(s)
I20260925 13:14:49.747905 14094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:49.748924 14162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:49.751286 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap starting.
I20260925 13:14:49.751869 14162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:49.752164 14162 log.cc:824] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:49.752857 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: No bootstrap required, opened a new log
I20260925 13:14:49.753867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14094
I20260925 13:14:49.753962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:14:49.754181 392 external_mini_cluster.cc:1428] Setting key c7f8ce9d27b93ddf086b56be107db58b
I20260925 13:14:49.754902 14162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:49.755100 14162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:49.755136 14162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364433dedec48fda122bb8fb5449f28, State: Initialized, Role: FOLLOWER
I20260925 13:14:49.755276 14162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:49.755362 14162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:14:49.755395 14162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:14:49.755483 14162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:49.756551 14162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:49.756657 14162 leader_election.cc:302] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9364433dedec48fda122bb8fb5449f28; no voters:
I20260925 13:14:49.756803 14162 leader_election.cc:288] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:14:49.756841 14167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:49.757018 14167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 LEADER]: Becoming Leader. State: Replica: 9364433dedec48fda122bb8fb5449f28, State: Running, Role: LEADER
I20260925 13:14:49.757081 14162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:49.757133 14167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:49.757455 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9364433dedec48fda122bb8fb5449f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260925 13:14:49.757589 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:49.757786 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9364433dedec48fda122bb8fb5449f28. Latest consensus state: current_term: 1 leader_uuid: "9364433dedec48fda122bb8fb5449f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260925 13:14:49.757884 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: This master's current role is: LEADER
W20260925 13:14:49.758899 14183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:14:49.758955 14183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:14:49.758998 14176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:49.759151 14176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:14:49.760737 14176 catalog_manager.cc:1347] Generated new cluster ID: 613390c31a2a476f8523f2a9cadd0528
I20260925 13:14:49.760792 14176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:14:49 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:49.765456 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:49.754749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.0.98.62:35321)
0925 13:14:49.754971 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:49.754981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:49.755014 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:14:49.755127 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:49.755135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:49.755193 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:49.755350 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:49.756025 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:49.757589 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:49.758841 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:49.759343 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:49.759476 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:49.762209 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:49.762214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:49.762214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:49.762225 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:49.763894 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:49.764276 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:49.764279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:49.764280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:49.764388 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:49.764607 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:49.764608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:49.764608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:49.764841 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:49.765042 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:49.765120 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 13:14:49.765275 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":113,"threads_started":1}
I20260925 13:14:49.790259 14176 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:14:49.791045 14176 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:14:49.795022 14176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Generated new TSK 0
I20260925 13:14:49.795209 14176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:14:49.822914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:49.905373 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:49.905596 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:49.905622 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:49.906829 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:49.906888 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:49.906915 14188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:14:49.908475 14188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:49.908730 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:49.908959 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:49.913964 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:49.913954 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:49.914019 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:49.914248 14188 server_base.cc:1034] running on GCE node
I20260925 13:14:49.914446 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:49.914695 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:49.915843 14188 hybrid_clock.cc:648] HybridClock initialized: now 1790342089915829 us; error 27 us; skew 500 ppm
Sep 25 13:14:49 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:49.918370 14188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:14:49.918844 14188 webserver.cc:460] Webserver started at http://127.0.98.1:34809/ using document root <none> and password file <none>
I20260925 13:14:49.919085 14188 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:49.919152 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:49.919301 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:49.920348 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "88b4646f6e5e4d0baba74d04de59a243"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "1454e2af5aefdedd5f22693996457166"
server_key_iv: "d9f56c4a2f5b3dffa2749d97517908a2"
server_key_version: "encryptionkey@0"
I20260925 13:14:49.920744 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88b4646f6e5e4d0baba74d04de59a243"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "1454e2af5aefdedd5f22693996457166"
server_key_iv: "d9f56c4a2f5b3dffa2749d97517908a2"
server_key_version: "encryptionkey@0"
I20260925 13:14:49.922153 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:14:49.922923 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:49.923132 14188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:49.923240 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "88b4646f6e5e4d0baba74d04de59a243"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "1454e2af5aefdedd5f22693996457166"
server_key_iv: "d9f56c4a2f5b3dffa2749d97517908a2"
server_key_version: "encryptionkey@0"
I20260925 13:14:49.923314 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:49.934960 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:49.936857 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:49.937024 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:49.937263 14188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:49.937592 14188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:49.937667 14188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:49.937804 14188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:49.937831 14188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:49.943722 14188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42407
I20260925 13:14:49.943771 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42407 every 8 connection(s)
I20260925 13:14:49.944139 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:14:49.948405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14188
I20260925 13:14:49.948544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:14:49.948772 392 external_mini_cluster.cc:1428] Setting key 3e7ec88570c5f4f775084313bc6f5b4c
Sep 25 13:14:49 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:49.955232 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:49.944904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55407 (local address 127.0.98.62:35321)
0925 13:14:49.945000 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:49.945003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:49.945199 (+ 196us) server_negotiation.cc:407] Connection header received
0925 13:14:49.945777 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:49.945792 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:49.945804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:49.945861 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:49.946840 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:49.947365 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:49.948875 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:49.949137 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:49.949211 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:49.951866 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:49.951867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:49.951867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:49.951875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:49.953620 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:49.954106 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:49.954107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:49.954108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:49.954180 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:49.954492 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:49.954493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:49.954493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:49.954613 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:49.954665 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:49.955079 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 13:14:49.955152 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:14:49.955581 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:49.944862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:55407)
0925 13:14:49.945058 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:49.945106 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:49.945639 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:49.945934 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:49.945939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:49.946531 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:49.946732 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:49.946737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:49.947474 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:49.947475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:49.948760 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:49.948764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:49.949228 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:49.949230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:49.949354 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:49.950003 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:49.950012 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:49.951737 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:49.953753 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:49.953762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:49.953765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:49.954025 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:49.954260 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:49.954261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:49.954261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:49.954421 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:49.954682 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:49.954691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:49.954898 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 13:14:49.955062 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 13:14:49.955282 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260925 13:14:49.956140 14317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260925 13:14:49.956274 14317 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:49.956471 14317 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260925 13:14:49.957113 14126 ts_manager.cc:194] Registered new tserver with Master: 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407)
I20260925 13:14:49.957904 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55407
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:14:49.976886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:50.062413 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:50.062623 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:50.062647 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:50.063756 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:50.063791 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:50.063815 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:14:50.065215 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:50.065402 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:50.065593 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:50.070449 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.070420 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.070545 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:50.070708 14322 server_base.cc:1034] running on GCE node
I20260925 13:14:50.070916 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:50.071122 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:50.072265 14322 hybrid_clock.cc:648] HybridClock initialized: now 1790342090072248 us; error 34 us; skew 500 ppm
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:50.074903 14322 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:14:50.075392 14322 webserver.cc:460] Webserver started at http://127.0.98.2:33515/ using document root <none> and password file <none>
I20260925 13:14:50.075639 14322 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:50.075707 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:50.075857 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:50.076921 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "8486fc0c411a4869a0f7303314260afa"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "f3740044c77e256b487a1f3e47fe5ffa"
server_key_iv: "2033d630dddef33a7d8634b55824f1ba"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.077281 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8486fc0c411a4869a0f7303314260afa"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "f3740044c77e256b487a1f3e47fe5ffa"
server_key_iv: "2033d630dddef33a7d8634b55824f1ba"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.078701 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:14:50.079453 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.079628 14322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:14:50.079733 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "8486fc0c411a4869a0f7303314260afa"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "f3740044c77e256b487a1f3e47fe5ffa"
server_key_iv: "2033d630dddef33a7d8634b55824f1ba"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.079807 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:50.088649 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:50.090977 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:50.091140 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:50.091361 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:50.091667 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:50.091722 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.091764 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:50.091792 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.097713 14322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37121
I20260925 13:14:50.097798 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37121 every 8 connection(s)
I20260925 13:14:50.098125 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:14:50.102746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14322
I20260925 13:14:50.102872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:14:50.103112 392 external_mini_cluster.cc:1428] Setting key d95e2a6eed540f41625035146dd475d0
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:50.107618 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.098872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54085 (local address 127.0.98.62:35321)
0925 13:14:50.098915 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.098918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.099098 (+ 180us) server_negotiation.cc:407] Connection header received
0925 13:14:50.099580 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.099582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.099592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.099641 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:50.100521 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.101025 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.102062 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.102241 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.102291 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.104704 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:50.104705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:50.104706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:50.104714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:50.106182 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.106618 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.106619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.106620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.106681 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.106946 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.106947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.106947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.107065 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:50.107140 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.107431 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.107529 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:14:50.107823 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.098850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:54085)
0925 13:14:50.098987 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.099023 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.099490 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.099693 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.099696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.100242 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:50.100434 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.100439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.101104 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.101104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.101985 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.101987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.102311 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.102311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.102407 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.103005 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:50.103018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:50.104601 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:50.106286 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.106292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.106295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.106529 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.106759 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.106760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.106761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.106870 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.107133 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:50.107139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:50.107283 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.107409 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.107570 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260925 13:14:50.108358 14451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260925 13:14:50.108474 14451 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:50.108692 14451 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260925 13:14:50.109110 14126 ts_manager.cc:194] Registered new tserver with Master: 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121)
I20260925 13:14:50.109496 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54085
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:14:50.128218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:50.209167 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:50.209373 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:50.209396 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:50.210625 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:50.210672 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:50.210698 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:14:50.212131 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:50.212383 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:50.212656 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:50.217557 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.217581 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.217583 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:50.217861 14456 server_base.cc:1034] running on GCE node
I20260925 13:14:50.218094 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:50.218331 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:50.219470 14456 hybrid_clock.cc:648] HybridClock initialized: now 1790342090219450 us; error 37 us; skew 500 ppm
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:50.221915 14456 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:14:50.222386 14456 webserver.cc:460] Webserver started at http://127.0.98.3:45945/ using document root <none> and password file <none>
I20260925 13:14:50.222681 14456 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:50.222748 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:50.222883 14456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:14:50.223927 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "a70d0bb08ec14c7ba294e396247795b6"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "6a319ed326cded926a2f7e1bdd615754"
server_key_iv: "bb8dc9c04f8d7dc9d2940ce72a5f74ea"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.224289 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a70d0bb08ec14c7ba294e396247795b6"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "6a319ed326cded926a2f7e1bdd615754"
server_key_iv: "bb8dc9c04f8d7dc9d2940ce72a5f74ea"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.225600 14456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:14:50.226462 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.226661 14456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:50.226759 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "a70d0bb08ec14c7ba294e396247795b6"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "6a319ed326cded926a2f7e1bdd615754"
server_key_iv: "bb8dc9c04f8d7dc9d2940ce72a5f74ea"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.226845 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:50.243281 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:50.245270 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:50.245428 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:50.245647 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:50.245985 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:50.246039 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.246073 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:50.246099 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.251616 14456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33897
I20260925 13:14:50.251659 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33897 every 8 connection(s)
I20260925 13:14:50.251964 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:14:50.253142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14456
I20260925 13:14:50.253253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:14:50.253418 392 external_mini_cluster.cc:1428] Setting key 401bb4f90ce7c7b840055431f74b7d7e
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:50.261193 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.252712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35657 (local address 127.0.98.62:35321)
0925 13:14:50.252781 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.252783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.252987 (+ 204us) server_negotiation.cc:407] Connection header received
0925 13:14:50.253549 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.253550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.253562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.253624 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:50.254730 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.255221 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.256199 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.256386 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.256479 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.258722 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:50.258724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:50.258724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:50.258732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:50.259879 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.260261 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.260262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.260263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.260321 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.260555 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.260556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.260557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.260673 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:50.260765 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.261033 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.261104 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:14:50.261368 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.252671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:35657)
0925 13:14:50.252866 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.252900 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.253427 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.253740 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.253744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.254438 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:50.254628 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.254633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.255312 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.255313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.256102 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.256105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.256501 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.256502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.256624 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.257069 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:50.257076 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:50.258632 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:50.259978 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.259985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.259987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.260188 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.260387 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.260388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.260388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.260490 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.260755 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:50.260761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:50.260893 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.261001 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.261149 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260925 13:14:50.261814 14585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260925 13:14:50.261909 14585 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:50.262076 14585 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260925 13:14:50.262415 14126 ts_manager.cc:194] Registered new tserver with Master: a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
I20260925 13:14:50.262741 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35657
I20260925 13:14:50.265107 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:14:50.271521 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.266350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.0.98.62:35321)
0925 13:14:50.266383 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.266385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.266499 (+ 114us) server_negotiation.cc:407] Connection header received
0925 13:14:50.266533 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.266534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.266541 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.266585 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:50.267103 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.267559 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.268245 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.268404 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.268456 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.269502 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:50.269503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:50.269503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:50.269507 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:50.270635 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.270942 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.270943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.270943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.270989 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.271120 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.271120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.271120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.271238 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:50.271308 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.271406 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.271465 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 13:14:50.273119 14126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:14:50.274679 14126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:14:50.280629 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.276646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37121 (local address 127.0.0.1:49034)
0925 13:14:50.276761 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.276797 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.276863 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.277173 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.277176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.277179 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:50.277240 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.277242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.278605 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.278606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.279390 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.279392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.280297 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.280298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.280387 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.280399 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.280464 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.280516 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260925 13:14:50.280864 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.276739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.0.98.2:37121)
0925 13:14:50.276921 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.276926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.276938 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:14:50.276980 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.276982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.276999 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.277083 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:50.277309 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.278497 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.279524 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.280142 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.280211 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.280263 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.280744 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.280789 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260925 13:14:50.280890 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.276662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33897 (local address 127.0.0.1:36558)
0925 13:14:50.276977 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.276986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.277025 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.277386 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.277387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.277390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:50.277423 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.277424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.278462 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.278463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.279751 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.279754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.280625 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.280626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.280708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.280716 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.280759 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.280806 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":46,"thread_start_us":59,"threads_started":1}
I20260925 13:14:50.280906 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.276819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.98.3:33897)
0925 13:14:50.277235 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.277239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.277249 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:14:50.277284 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.277287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.277301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.277378 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:50.277506 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.278340 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.279839 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.280529 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.280702 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.280769 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.280791 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.280827 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":82,"threads_started":1}
I20260925 13:14:50.281291 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.276860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42407 (local address 127.0.0.1:42998)
0925 13:14:50.277070 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.277081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.277123 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.277483 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.277484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.277486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:50.277534 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.277535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.278752 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.278753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.279808 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.279810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.280540 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.280541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.280610 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.280613 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.280671 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.281228 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260925 13:14:50.281309 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.277089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.0.98.1:42407)
0925 13:14:50.277255 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.277259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.277269 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:14:50.277309 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.277311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.277325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.277405 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:50.277612 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.278658 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.279893 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.280436 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.280529 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.280590 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.281190 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.281234 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260925 13:14:50.281618 14385 tablet_service.cc:1467] Processing CreateTablet for tablet cc582bb59bd148ef874b1e39afc1df42 (DEFAULT_TABLE table=test-table [id=9bf4048df4e0423da4be85bdee391dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:50.281618 14519 tablet_service.cc:1467] Processing CreateTablet for tablet cc582bb59bd148ef874b1e39afc1df42 (DEFAULT_TABLE table=test-table [id=9bf4048df4e0423da4be85bdee391dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:50.281986 14385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc582bb59bd148ef874b1e39afc1df42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:50.281986 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc582bb59bd148ef874b1e39afc1df42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:50.282142 14251 tablet_service.cc:1467] Processing CreateTablet for tablet cc582bb59bd148ef874b1e39afc1df42 (DEFAULT_TABLE table=test-table [id=9bf4048df4e0423da4be85bdee391dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:50.282464 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc582bb59bd148ef874b1e39afc1df42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:50.285243 14601 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Bootstrap starting.
I20260925 13:14:50.285398 14599 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap starting.
I20260925 13:14:50.285830 14600 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Bootstrap starting.
I20260925 13:14:50.285931 14601 tablet_bootstrap.cc:654] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:50.286041 14599 tablet_bootstrap.cc:654] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:50.286504 14599 log.cc:824] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:50.286523 14601 log.cc:824] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:50.286567 14600 tablet_bootstrap.cc:654] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:50.287117 14600 log.cc:824] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:50.287300 14599 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: No bootstrap required, opened a new log
I20260925 13:14:50.287365 14599 ts_tablet_manager.cc:1397] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:50.287830 14600 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: No bootstrap required, opened a new log
I20260925 13:14:50.287899 14600 ts_tablet_manager.cc:1397] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:14:50.289314 14599 raft_consensus.cc:348] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.289530 14599 raft_consensus.cc:374] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:50.289569 14599 raft_consensus.cc:729] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4646f6e5e4d0baba74d04de59a243, State: Initialized, Role: FOLLOWER
I20260925 13:14:50.289722 14599 consensus_queue.cc:260] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.289765 14600 raft_consensus.cc:348] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.289927 14600 raft_consensus.cc:374] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:50.290006 14600 raft_consensus.cc:729] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70d0bb08ec14c7ba294e396247795b6, State: Initialized, Role: FOLLOWER
I20260925 13:14:50.290019 14599 ts_tablet_manager.cc:1428] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 13:14:50.290063 14317 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260925 13:14:50.290122 14600 consensus_queue.cc:260] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.290381 14600 ts_tablet_manager.cc:1428] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:14:50.290409 14601 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: No bootstrap required, opened a new log
I20260925 13:14:50.290472 14601 ts_tablet_manager.cc:1397] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 13:14:50.290513 14585 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260925 13:14:50.292047 14601 raft_consensus.cc:348] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.292178 14601 raft_consensus.cc:374] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:50.292198 14601 raft_consensus.cc:729] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8486fc0c411a4869a0f7303314260afa, State: Initialized, Role: FOLLOWER
I20260925 13:14:50.292268 14601 consensus_queue.cc:260] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.292456 14601 ts_tablet_manager.cc:1428] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:14:50.292503 14451 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
W20260925 13:14:50.348824 14452 tablet.cc:2369] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:14:50.444885 14318 tablet.cc:2369] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:14:50.502573 14586 tablet.cc:2369] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:14:50.590201 14607 raft_consensus.cc:482] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:50.590320 14607 raft_consensus.cc:504] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.590669 14607 leader_election.cc:288] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897), 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407)
I20260925 13:14:50.593474 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.591014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49395 (local address 127.0.98.3:33897)
0925 13:14:50.591082 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.591083 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.591091 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:14:50.591115 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.591115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.591124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.591193 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:50.591347 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.591811 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.592744 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.593124 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.593209 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.593220 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.593367 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.593407 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:14:50.593483 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.590883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33897 (local address 127.0.98.2:49395)
0925 13:14:50.590940 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.590951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.590995 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.591197 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.591198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.591199 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:50.591274 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.591276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.591905 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.591906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.592650 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.592652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.593207 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.593208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.593290 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.593293 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.593360 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.593402 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:14:50.593694 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.591022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37877 (local address 127.0.98.1:42407)
0925 13:14:50.591068 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.591070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.591218 (+ 148us) server_negotiation.cc:407] Connection header received
0925 13:14:50.591292 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.591293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.591312 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.591371 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:50.591530 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.592015 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.592939 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.593298 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.593379 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.593388 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.593537 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.593587 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:14:50.593869 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.590937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42407 (local address 127.0.98.2:37877)
0925 13:14:50.591128 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.591137 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.591226 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.591375 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.591376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.591377 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:50.591456 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.591458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.592101 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.592102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.592843 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.592845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.593382 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.593383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.593460 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.593464 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.593656 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.593719 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260925 13:14:50.593850 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc582bb59bd148ef874b1e39afc1df42" candidate_uuid: "8486fc0c411a4869a0f7303314260afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a70d0bb08ec14c7ba294e396247795b6" is_pre_election: true
I20260925 13:14:50.594012 14539 raft_consensus.cc:2454] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8486fc0c411a4869a0f7303314260afa in term 0.
I20260925 13:14:50.594213 14271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc582bb59bd148ef874b1e39afc1df42" candidate_uuid: "8486fc0c411a4869a0f7303314260afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4646f6e5e4d0baba74d04de59a243" is_pre_election: true
I20260925 13:14:50.594362 14271 raft_consensus.cc:2454] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8486fc0c411a4869a0f7303314260afa in term 0.
I20260925 13:14:50.594525 14341 leader_election.cc:302] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6; no voters:
I20260925 13:14:50.594666 14607 raft_consensus.cc:2781] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:14:50.594720 14607 raft_consensus.cc:482] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:14:50.594769 14607 raft_consensus.cc:3037] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:50.595468 14607 raft_consensus.cc:504] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.595619 14607 leader_election.cc:288] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [CANDIDATE]: Term 1 election: Requested vote from peers a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897), 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407)
I20260925 13:14:50.595746 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc582bb59bd148ef874b1e39afc1df42" candidate_uuid: "8486fc0c411a4869a0f7303314260afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a70d0bb08ec14c7ba294e396247795b6"
I20260925 13:14:50.595820 14271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc582bb59bd148ef874b1e39afc1df42" candidate_uuid: "8486fc0c411a4869a0f7303314260afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4646f6e5e4d0baba74d04de59a243"
I20260925 13:14:50.595853 14539 raft_consensus.cc:3037] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:50.595884 14271 raft_consensus.cc:3037] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:50.596774 14271 raft_consensus.cc:2454] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8486fc0c411a4869a0f7303314260afa in term 1.
I20260925 13:14:50.596774 14539 raft_consensus.cc:2454] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8486fc0c411a4869a0f7303314260afa in term 1.
I20260925 13:14:50.596971 14338 leader_election.cc:302] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8486fc0c411a4869a0f7303314260afa, 88b4646f6e5e4d0baba74d04de59a243; no voters:
I20260925 13:14:50.597092 14607 raft_consensus.cc:2781] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:50.597276 14607 raft_consensus.cc:686] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 1 LEADER]: Becoming Leader. State: Replica: 8486fc0c411a4869a0f7303314260afa, State: Running, Role: LEADER
I20260925 13:14:50.597400 14607 consensus_queue.cc:237] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.598424 14125 catalog_manager.cc:5495] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8486fc0c411a4869a0f7303314260afa (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8486fc0c411a4869a0f7303314260afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } health_report { overall_health: UNKNOWN } } }
I20260925 13:14:50.633638 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.631448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.98.2:37121)
0925 13:14:50.631498 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.631499 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.631507 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:14:50.631546 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.631547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.631566 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.631628 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:14:50.631750 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.632285 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.633006 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.633192 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.633240 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.633373 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:14:50.633470 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:14:50.633530 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.633562 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.633579 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:14:50.635365 14539 raft_consensus.cc:1261] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Refusing update from remote peer 8486fc0c411a4869a0f7303314260afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:50.635394 14271 raft_consensus.cc:1261] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Refusing update from remote peer 8486fc0c411a4869a0f7303314260afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:50.635795 14609 consensus_queue.cc:1035] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:50.635937 14607 consensus_queue.cc:1035] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:50.639238 14614 mvcc.cc:205] Tried to move back new op lower bound from 7333241203239882752 to 7333241203087941632. Current Snapshot: MvccSnapshot[applied={T|T < 7333241203239882752 or (T in {7333241203239882752})}]
I20260925 13:14:50.640403 14124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:14:50.640496 14124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:14:50.641280 14124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Sending DeleteTablet for 3 replicas of tablet cc582bb59bd148ef874b1e39afc1df42
I20260925 13:14:50.641616 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet cc582bb59bd148ef874b1e39afc1df42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260925 13:14:50.641770 14385 tablet_service.cc:1514] Processing DeleteTablet for tablet cc582bb59bd148ef874b1e39afc1df42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49034
I20260925 13:14:50.641850 14622 tablet_replica.cc:330] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: stopping tablet replica
I20260925 13:14:50.641954 14622 raft_consensus.cc:2229] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:14:50.641999 14623 tablet_replica.cc:330] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: stopping tablet replica
I20260925 13:14:50.642016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14188
I20260925 13:14:50.642066 14622 raft_consensus.cc:2258] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:14:50.642076 14623 raft_consensus.cc:2229] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:14:50.642915 14623 raft_consensus.cc:2258] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:14:50.643178 14622 ts_tablet_manager.cc:1905] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:14:50.643545 14623 ts_tablet_manager.cc:1905] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:14:50.644845 14622 ts_tablet_manager.cc:1918] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:14:50.644912 14622 log.cc:1196] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/cc582bb59bd148ef874b1e39afc1df42
I20260925 13:14:50.645180 14623 ts_tablet_manager.cc:1918] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:14:50.645207 14622 ts_tablet_manager.cc:1939] T cc582bb59bd148ef874b1e39afc1df42 P a70d0bb08ec14c7ba294e396247795b6: Deleting consensus metadata
I20260925 13:14:50.645239 14623 log.cc:1196] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/cc582bb59bd148ef874b1e39afc1df42
I20260925 13:14:50.645468 14623 ts_tablet_manager.cc:1939] T cc582bb59bd148ef874b1e39afc1df42 P 8486fc0c411a4869a0f7303314260afa: Deleting consensus metadata
I20260925 13:14:50.645707 14113 catalog_manager.cc:4841] TS a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): tablet cc582bb59bd148ef874b1e39afc1df42 (table test-table [id=9bf4048df4e0423da4be85bdee391dd3]) successfully deleted
I20260925 13:14:50.645895 14110 catalog_manager.cc:4841] TS 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): tablet cc582bb59bd148ef874b1e39afc1df42 (table test-table [id=9bf4048df4e0423da4be85bdee391dd3]) successfully deleted
W20260925 13:14:50.648182 14110 connection.cc:535] client connection to 127.0.98.1:42407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 13:14:50.648269 14110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 13:14:50.648792 14597 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:14:50.648595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42407 (local address 127.0.0.1:43000)
0925 13:14:50.648652 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.648663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.648701 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.648762 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42407: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 13:14:50.648934 14110 catalog_manager.cc:4584] TS 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc582bb59bd148ef874b1e39afc1df42: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42407: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 13:14:50.649031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14322
I20260925 13:14:50.654031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14456
I20260925 13:14:50.658774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14094
I20260925 13:14:50.664068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--webserver_interface=127.0.98.62
--webserver_port=37701
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:50.744001 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:50.744199 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:50.744221 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:50.745374 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:50.745415 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:50.745429 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:50.745442 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:50.745503 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:50.746999 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:50.747251 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:50.747500 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:50.755877 14632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:50.756048 14625 server_base.cc:1034] running on GCE node
W20260925 13:14:50.755918 14634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.755877 14631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:50.756363 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:50.756716 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:50.757877 14625 hybrid_clock.cc:648] HybridClock initialized: now 1790342090757865 us; error 24 us; skew 500 ppm
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:50.771695 14625 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:50.772202 14625 webserver.cc:460] Webserver started at http://127.0.98.62:37701/ using document root <none> and password file <none>
I20260925 13:14:50.772441 14625 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:50.772531 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:50.773976 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:50.774627 14641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.774778 14625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:50.774876 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "9364433dedec48fda122bb8fb5449f28"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "edd2e4b70d9317f522417c943a579fa1"
server_key_iv: "ad0135cdca8c2cbb75224c266c310dc7"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.775245 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:50.798345 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:50.800782 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:50.800953 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:50.805159 14625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35321
I20260925 13:14:50.805205 14693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35321 every 8 connection(s)
I20260925 13:14:50.805655 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:50.807272 14694 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:14:50.807863 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap starting.
I20260925 13:14:50.808986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14625
I20260925 13:14:50.809275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42407
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35321
--webserver_port=34809
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:50.816291 14694 log.cc:824] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:50.818357 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:14:50.818651 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap complete.
I20260925 13:14:50.821023 14694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:50.821280 14694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364433dedec48fda122bb8fb5449f28, State: Initialized, Role: FOLLOWER
I20260925 13:14:50.821636 14694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:50.821771 14694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:14:50.821834 14694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:14:50.821898 14694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:14:50.822927 14694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:50.823047 14694 leader_election.cc:302] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9364433dedec48fda122bb8fb5449f28; no voters:
I20260925 13:14:50.823257 14694 leader_election.cc:288] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:14:50.823315 14698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:14:50.823458 14698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 LEADER]: Becoming Leader. State: Replica: 9364433dedec48fda122bb8fb5449f28, State: Running, Role: LEADER
I20260925 13:14:50.823561 14694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:50.823657 14698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:50.823997 14699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9364433dedec48fda122bb8fb5449f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260925 13:14:50.824100 14699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:50.823998 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9364433dedec48fda122bb8fb5449f28. Latest consensus state: current_term: 2 leader_uuid: "9364433dedec48fda122bb8fb5449f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260925 13:14:50.824275 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:50.824666 14706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:50.825412 14706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:14:50.825506 14706 catalog_manager.cc:1259] Loaded cluster ID: 613390c31a2a476f8523f2a9cadd0528
I20260925 13:14:50.825538 14706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:14:50.825826 14706 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 13:14:50.830649 14706 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 13:14:50.833127 14706 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:14:50.833395 14706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Loaded TSK: 0
I20260925 13:14:50.833729 14706 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:14:50.892203 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:50.892398 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:50.892421 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:50.893734 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:50.893779 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:50.893806 14696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:14:50.895244 14696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34809
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:50.895488 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:50.895737 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:50.901522 14724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.901551 14722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:50.901520 14721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:50.901777 14696 server_base.cc:1034] running on GCE node
I20260925 13:14:50.901957 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:50.902221 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:50.903374 14696 hybrid_clock.cc:648] HybridClock initialized: now 1790342090903358 us; error 33 us; skew 500 ppm
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:50.905923 14696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:14:50.906420 14696 webserver.cc:460] Webserver started at http://127.0.98.1:34809/ using document root <none> and password file <none>
I20260925 13:14:50.906661 14696 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:50.906718 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:50.908061 14696 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:50.908751 14731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:50.908921 14696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:50.909025 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "88b4646f6e5e4d0baba74d04de59a243"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "1454e2af5aefdedd5f22693996457166"
server_key_iv: "d9f56c4a2f5b3dffa2749d97517908a2"
server_key_version: "encryptionkey@0"
I20260925 13:14:50.909368 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:50.921993 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:50.924371 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:50.924533 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:50.924824 14696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:50.925360 14738 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:14:50.927256 14696 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:14:50.927318 14696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:14:50.927361 14696 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:14:50.927969 14696 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:14:50.928030 14696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:14:50.928097 14738 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap starting.
I20260925 13:14:50.935171 14696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42407
I20260925 13:14:50.935552 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:14:50.936101 14845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42407 every 8 connection(s)
I20260925 13:14:50.939236 14738 log.cc:824] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Log is configured to *not* fsync() on all Append() calls
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:50.943936 14738 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:14:50.944244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14696
I20260925 13:14:50.944312 14738 tablet_bootstrap.cc:492] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap complete.
I20260925 13:14:50.944537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37121
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35321
--webserver_port=33515
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:50.946177 14738 ts_tablet_manager.cc:1397] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.010s
I20260925 13:14:50.947803 14738 raft_consensus.cc:348] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.948204 14738 raft_consensus.cc:729] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4646f6e5e4d0baba74d04de59a243, State: Initialized, Role: FOLLOWER
I20260925 13:14:50.948378 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.936331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:43915)
0925 13:14:50.936543 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.936591 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.937054 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.938188 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.938192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.938486 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:50.938672 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.938678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.939875 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.939876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.940681 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.940683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.941307 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.941308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.941438 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.942050 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:50.942071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:50.943908 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:50.946571 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.946577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.946578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.946863 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.947139 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.947140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.947140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.947271 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.947727 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:50.947732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:50.947866 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.947954 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.948120 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260925 13:14:50.948359 14738 consensus_queue.cc:260] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:50.948642 14738 ts_tablet_manager.cc:1428] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 13:14:50.949121 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.936415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43915 (local address 127.0.98.62:35321)
0925 13:14:50.937813 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.937820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.937848 (+ 28us) server_negotiation.cc:407] Connection header received
0925 13:14:50.937954 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.937959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.938013 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.938159 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:50.938798 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.939756 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.940781 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.941195 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.941322 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.944104 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:50.944110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:50.944110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:50.944125 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:50.946460 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.946961 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.946964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.946964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.947073 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.947360 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.947361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.947361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.947645 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:50.948720 (+ 1075us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.948796 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.948933 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":114,"threads_started":1}
I20260925 13:14:50.949760 14846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260925 13:14:50.949879 14846 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:50.950078 14846 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260925 13:14:50.950924 14658 ts_manager.cc:194] Registered new tserver with Master: 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407)
I20260925 13:14:50.952181 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43915
I20260925 13:14:50.953270 14846 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
Sep 25 13:14:50 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:14:50.962013 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.951558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42407 (local address 127.0.0.1:43008)
0925 13:14:50.952503 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:50.952537 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:50.952599 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:50.952928 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:50.952932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:50.953242 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:50.953299 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.953301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.954790 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.954792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.955744 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:50.955747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.956403 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.956407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:50.956473 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.957072 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:50.957076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:50.958811 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:50.960806 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.960813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.960815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.961057 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.961271 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:50.961272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:50.961272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:50.961326 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:50.961640 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:50.961645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:50.961730 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 13:14:50.961793 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:14:50.961904 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":96,"threads_started":1}
I20260925 13:14:50.962483 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:50.951820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.98.1:42407)
0925 13:14:50.952743 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:50.952748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:50.952757 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:14:50.952791 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:50.952792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:50.952805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:50.952865 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:50.953406 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.954675 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.955841 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:50.956300 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:50.956684 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:50.958905 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:50.958912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:50.958912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:50.958919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:50.960703 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.961144 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.961147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.961147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.961205 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:50.961403 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:50.961403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:50.961404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:50.961569 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:50.962232 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:50.962281 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 13:14:50.962387 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":77,"threads_started":1}
I20260925 13:14:50.962771 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet cc582bb59bd148ef874b1e39afc1df42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43008
I20260925 13:14:50.963428 14855 tablet_replica.cc:330] stopping tablet replica
I20260925 13:14:50.963541 14855 raft_consensus.cc:2229] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:14:50.963608 14855 raft_consensus.cc:2258] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:14:50.963924 14855 ts_tablet_manager.cc:1905] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:14:50.966207 14855 ts_tablet_manager.cc:1918] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:14:50.966284 14855 log.cc:1196] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/cc582bb59bd148ef874b1e39afc1df42
I20260925 13:14:50.966498 14855 ts_tablet_manager.cc:1939] T cc582bb59bd148ef874b1e39afc1df42 P 88b4646f6e5e4d0baba74d04de59a243: Deleting consensus metadata
I20260925 13:14:50.967132 14642 catalog_manager.cc:4841] TS 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407): tablet cc582bb59bd148ef874b1e39afc1df42 (table test-table [id=9bf4048df4e0423da4be85bdee391dd3]) successfully deleted
W20260925 13:14:51.032400 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:51.032625 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:51.032650 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:51.033793 14851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:51.033833 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:51.033859 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:14:51.035231 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33515
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:51.035413 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:51.035585 14851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:51.040349 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.040333 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.040412 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.040522 14851 server_base.cc:1034] running on GCE node
I20260925 13:14:51.040829 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:51.041077 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:51.042232 14851 hybrid_clock.cc:648] HybridClock initialized: now 1790342091042231 us; error 35 us; skew 500 ppm
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:51.044549 14851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:14:51.044956 14851 webserver.cc:460] Webserver started at http://127.0.98.2:33515/ using document root <none> and password file <none>
I20260925 13:14:51.045194 14851 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:51.045274 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:51.046633 14851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.047192 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.047353 14851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:51.047456 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "8486fc0c411a4869a0f7303314260afa"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "f3740044c77e256b487a1f3e47fe5ffa"
server_key_iv: "2033d630dddef33a7d8634b55824f1ba"
server_key_version: "encryptionkey@0"
I20260925 13:14:51.047792 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:51.057183 14851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:51.058945 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:51.059099 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:51.059317 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:51.059625 14851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:51.059679 14851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.059721 14851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:51.059746 14851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.065502 14851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37121
I20260925 13:14:51.065555 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37121 every 8 connection(s)
I20260925 13:14:51.065946 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:14:51.070710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14851
I20260925 13:14:51.071054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33897
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35321
--webserver_port=45945
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:51.075695 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.066699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51833 (local address 127.0.98.62:35321)
0925 13:14:51.066762 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.066765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.066982 (+ 217us) server_negotiation.cc:407] Connection header received
0925 13:14:51.067446 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.067447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.067462 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.067529 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:51.068460 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.068943 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.069948 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.070137 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.070188 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.072796 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:51.072798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:51.072798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:51.072804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:51.074263 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:51.074761 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:51.074761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:51.074762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:51.074810 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:51.075084 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:51.075085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:51.075085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:51.075179 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:51.075229 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.075565 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.075632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:14:51.075997 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.066672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:51833)
0925 13:14:51.066831 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.066889 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.067360 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.067612 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:51.067616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:51.068190 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:51.068376 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.068381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.069025 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.069026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.069879 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.069882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.070206 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.070207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.070296 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.070881 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:51.070891 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:51.072700 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:51.074346 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:51.074353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:51.074355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:51.074687 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:51.074877 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:51.074878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:51.074879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:51.075017 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:51.075244 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:51.075250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:51.075422 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 13:14:51.075550 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 13:14:51.075727 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260925 13:14:51.076537 14984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260925 13:14:51.076640 14984 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:51.076838 14984 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260925 13:14:51.077267 14658 ts_manager.cc:194] Registered new tserver with Master: 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121)
I20260925 13:14:51.077637 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51833
W20260925 13:14:51.153321 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:51.153518 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:51.153541 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:51.154767 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:51.154814 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:51.154839 14987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:14:51.156294 14987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45945
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:51.156531 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:51.156777 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:51.161729 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.161779 14987 server_base.cc:1034] running on GCE node
W20260925 13:14:51.161738 14993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.161733 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.162141 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:51.162384 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:51.163525 14987 hybrid_clock.cc:648] HybridClock initialized: now 1790342091163504 us; error 36 us; skew 500 ppm
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:51.166311 14987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:14:51.166796 14987 webserver.cc:460] Webserver started at http://127.0.98.3:45945/ using document root <none> and password file <none>
I20260925 13:14:51.167039 14987 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:51.167096 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:51.168499 14987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.169068 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.169241 14987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:51.169344 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "a70d0bb08ec14c7ba294e396247795b6"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "6a319ed326cded926a2f7e1bdd615754"
server_key_iv: "bb8dc9c04f8d7dc9d2940ce72a5f74ea"
server_key_version: "encryptionkey@0"
I20260925 13:14:51.169720 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:51.185771 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:51.187827 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:51.187988 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:51.188230 14987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:51.188568 14987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:51.188601 14987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.188624 14987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:51.188671 14987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.194268 14987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33897
I20260925 13:14:51.194340 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33897 every 8 connection(s)
I20260925 13:14:51.194685 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:14:51.196324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 14987
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:14:51.203418 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.196749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.98.62:35321)
0925 13:14:51.196913 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.196915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.196926 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:14:51.196948 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.196949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.196959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.197015 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:51.197650 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.198241 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.198954 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.199227 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.199275 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.200524 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:51.200526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:51.200526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:51.200533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:51.202258 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:51.202626 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:51.202627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:51.202627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:51.202679 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:51.202882 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:51.202883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:51.202884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:51.203022 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:51.203173 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.203263 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.203344 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260925 13:14:51.205049 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.195443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60523 (local address 127.0.98.62:35321)
0925 13:14:51.195493 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.195495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.195701 (+ 206us) server_negotiation.cc:407] Connection header received
0925 13:14:51.196189 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.196191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.196202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.196284 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:51.197431 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.198040 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.199140 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.199354 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.199421 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.202153 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:51.202154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:51.202154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:51.202160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:51.203762 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:51.204225 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:51.204226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:51.204226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:51.204286 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:51.204538 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:51.204538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:51.204538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:51.204632 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:51.204679 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.204926 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.204990 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:14:51.205284 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.195400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:60523)
0925 13:14:51.195568 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.195609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.196076 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.196357 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:51.196361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:51.197112 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:51.197321 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.197326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.198124 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.198125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.199055 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.199058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.199434 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.199435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.199576 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.200075 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:51.200083 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:51.202052 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:51.203840 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:51.203846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:51.203849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:51.204153 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:51.204358 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:51.204358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:51.204359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:51.204478 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:51.204687 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:51.204692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:51.204802 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 13:14:51.204883 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:14:51.205027 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260925 13:14:51.205745 15116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260925 13:14:51.205861 15116 heartbeater.cc:464] Registering TS with master...
I20260925 13:14:51.206043 15116 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260925 13:14:51.206431 14658 ts_manager.cc:194] Registered new tserver with Master: a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
I20260925 13:14:51.206751 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60523
I20260925 13:14:51.215252 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:14:51.218005 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.215619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.98.62:35321)
0925 13:14:51.215678 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.215679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.215823 (+ 144us) server_negotiation.cc:407] Connection header received
0925 13:14:51.215902 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.215903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.215933 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.215992 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:14:51.216109 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.216585 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.217297 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.217483 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.217530 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.217719 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:14:51.217816 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:14:51.217884 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.217940 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.217954 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:14:51.218267 14658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:14:51.219861 14658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:14:51.222103 14780 tablet_service.cc:1467] Processing CreateTablet for tablet e5bcc8de9204491dadc394e3a72a220e (DEFAULT_TABLE table=test-table [id=74d6c3282d1f4b659f8820dc9847cdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:51.222311 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bcc8de9204491dadc394e3a72a220e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:51.223929 14738 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap starting.
I20260925 13:14:51.224242 14738 tablet_bootstrap.cc:654] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:51.225260 14738 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: No bootstrap required, opened a new log
I20260925 13:14:51.225322 14738 ts_tablet_manager.cc:1397] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:14:51.225495 14738 raft_consensus.cc:348] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.225567 14738 raft_consensus.cc:374] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:51.225625 14738 raft_consensus.cc:729] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4646f6e5e4d0baba74d04de59a243, State: Initialized, Role: FOLLOWER
I20260925 13:14:51.225664 14738 consensus_queue.cc:260] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.225765 14738 ts_tablet_manager.cc:1428] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:14:51.225871 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.221646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37121 (local address 127.0.0.1:49050)
0925 13:14:51.221737 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.221748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.221788 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.222483 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:51.222484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:51.222486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:51.222529 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.222531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.223587 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.223588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.224607 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.224610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.225557 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.225558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.225646 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.225665 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 13:14:51.225750 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 13:14:51.225796 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":6}
I20260925 13:14:51.225893 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.221773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.0.98.2:37121)
0925 13:14:51.222323 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.222327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.222336 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:14:51.222376 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.222378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.222395 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.222472 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:51.222587 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.223453 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.224694 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.225458 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.225554 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.225635 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.225766 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.225804 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":98,"threads_started":1}
I20260925 13:14:51.226537 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.221784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33897 (local address 127.0.0.1:36568)
0925 13:14:51.222139 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.222147 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.222186 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.222611 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:51.222613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:51.222615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:51.222711 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.222712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.223952 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.223953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.225235 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.225255 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.226280 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.226281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.226347 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.226351 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:14:51.226432 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:14:51.226471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":86,"threads_started":1}
I20260925 13:14:51.226572 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.221982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.0.98.3:33897)
0925 13:14:51.222172 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.222177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.222232 (+ 55us) server_negotiation.cc:407] Connection header received
0925 13:14:51.222275 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.222277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.222294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.222537 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:51.222787 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.223837 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.225347 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.226126 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.226192 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.226248 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.226450 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.226494 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260925 13:14:51.226550 14918 tablet_service.cc:1467] Processing CreateTablet for tablet e5bcc8de9204491dadc394e3a72a220e (DEFAULT_TABLE table=test-table [id=74d6c3282d1f4b659f8820dc9847cdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:51.226895 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bcc8de9204491dadc394e3a72a220e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:51.229593 15126 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Bootstrap starting.
I20260925 13:14:51.230163 15050 tablet_service.cc:1467] Processing CreateTablet for tablet e5bcc8de9204491dadc394e3a72a220e (DEFAULT_TABLE table=test-table [id=74d6c3282d1f4b659f8820dc9847cdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:14:51.230284 15126 tablet_bootstrap.cc:654] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:51.230501 15050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bcc8de9204491dadc394e3a72a220e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:14:51.230865 15126 log.cc:824] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:51.231637 15126 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: No bootstrap required, opened a new log
I20260925 13:14:51.231712 15126 ts_tablet_manager.cc:1397] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:14:51.233754 15126 raft_consensus.cc:348] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.233939 15126 raft_consensus.cc:374] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:51.233978 15126 raft_consensus.cc:729] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8486fc0c411a4869a0f7303314260afa, State: Initialized, Role: FOLLOWER
I20260925 13:14:51.234036 15128 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Bootstrap starting.
I20260925 13:14:51.234122 15126 consensus_queue.cc:260] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.234411 15126 ts_tablet_manager.cc:1428] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 13:14:51.234493 14984 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260925 13:14:51.234689 15128 tablet_bootstrap.cc:654] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Neither blocks nor log segments found. Creating new log.
I20260925 13:14:51.235330 15128 log.cc:824] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:51.236008 15128 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: No bootstrap required, opened a new log
I20260925 13:14:51.236063 15128 ts_tablet_manager.cc:1397] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:14:51.237349 15128 raft_consensus.cc:348] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.237499 15128 raft_consensus.cc:374] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:14:51.237522 15128 raft_consensus.cc:729] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70d0bb08ec14c7ba294e396247795b6, State: Initialized, Role: FOLLOWER
I20260925 13:14:51.237686 15128 consensus_queue.cc:260] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.237974 15128 ts_tablet_manager.cc:1428] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:14:51.237977 15116 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260925 13:14:51.308391 15131 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:51.308540 15131 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.308892 15131 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407)
I20260925 13:14:51.312064 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.309244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55999 (local address 127.0.98.2:37121)
0925 13:14:51.309304 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.309306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.309316 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:14:51.309348 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.309349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.309361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.309445 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:51.309659 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.310237 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.311271 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.311745 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.311802 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.311811 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.311979 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.312009 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:14:51.312095 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.309102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37121 (local address 127.0.98.3:55999)
0925 13:14:51.309164 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.309177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.309225 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.309457 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:51.309458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:51.309478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:51.309563 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.309565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.310315 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.310316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.311175 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.311178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.311823 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.311824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.311903 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.311907 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:14:51.311966 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:14:51.312017 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 13:14:51.312546 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.309228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42407 (local address 127.0.98.3:35035)
0925 13:14:51.309412 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.309422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.309483 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.309678 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:51.309679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:51.309681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:14:51.309749 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.309751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.310417 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.310418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.311392 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:51.311422 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.312293 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.312294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:51.312365 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.312369 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:14:51.312429 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:14:51.312468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
I20260925 13:14:51.312494 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bcc8de9204491dadc394e3a72a220e" candidate_uuid: "a70d0bb08ec14c7ba294e396247795b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8486fc0c411a4869a0f7303314260afa" is_pre_election: true
I20260925 13:14:51.312582 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.309307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35035 (local address 127.0.98.1:42407)
0925 13:14:51.309363 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.309365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.309522 (+ 157us) server_negotiation.cc:407] Connection header received
0925 13:14:51.309552 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.309553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.309579 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.309645 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:14:51.309820 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.310334 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.311544 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.312081 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.312146 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.312237 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.312449 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.312500 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:14:51.312685 14938 raft_consensus.cc:2454] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70d0bb08ec14c7ba294e396247795b6 in term 0.
I20260925 13:14:51.312956 15003 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6; no voters:
I20260925 13:14:51.313033 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bcc8de9204491dadc394e3a72a220e" candidate_uuid: "a70d0bb08ec14c7ba294e396247795b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4646f6e5e4d0baba74d04de59a243" is_pre_election: true
I20260925 13:14:51.313120 15131 raft_consensus.cc:2781] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:14:51.313197 15131 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:14:51.313202 14800 raft_consensus.cc:2454] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70d0bb08ec14c7ba294e396247795b6 in term 0.
I20260925 13:14:51.313231 15131 raft_consensus.cc:3037] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:51.314143 15131 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.314286 15131 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [CANDIDATE]: Term 1 election: Requested vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407)
I20260925 13:14:51.314456 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bcc8de9204491dadc394e3a72a220e" candidate_uuid: "a70d0bb08ec14c7ba294e396247795b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8486fc0c411a4869a0f7303314260afa"
I20260925 13:14:51.314455 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bcc8de9204491dadc394e3a72a220e" candidate_uuid: "a70d0bb08ec14c7ba294e396247795b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4646f6e5e4d0baba74d04de59a243"
I20260925 13:14:51.314520 14938 raft_consensus.cc:3037] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:51.314567 14800 raft_consensus.cc:3037] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:14:51.315208 14938 raft_consensus.cc:2454] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70d0bb08ec14c7ba294e396247795b6 in term 1.
I20260925 13:14:51.315286 14800 raft_consensus.cc:2454] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70d0bb08ec14c7ba294e396247795b6 in term 1.
I20260925 13:14:51.315357 15003 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6; no voters:
I20260925 13:14:51.315464 15131 raft_consensus.cc:2781] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:14:51.315595 15131 raft_consensus.cc:686] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 1 LEADER]: Becoming Leader. State: Replica: a70d0bb08ec14c7ba294e396247795b6, State: Running, Role: LEADER
I20260925 13:14:51.315665 15131 consensus_queue.cc:237] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
W20260925 13:14:51.316461 14985 tablet.cc:2369] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:14:51.316504 14658 catalog_manager.cc:5495] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a70d0bb08ec14c7ba294e396247795b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } health_report { overall_health: UNKNOWN } } }
I20260925 13:14:51.342077 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.339739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.98.3:33897)
0925 13:14:51.339815 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:51.339817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:51.339826 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:14:51.339867 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:51.339868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:51.339892 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:51.339961 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:14:51.340080 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.340576 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.341266 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:51.341457 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:51.341513 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:51.341638 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:14:51.341797 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:14:51.341869 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:51.341903 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:14:51.341923 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:14:51.344062 14938 raft_consensus.cc:1261] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Refusing update from remote peer a70d0bb08ec14c7ba294e396247795b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:51.344063 14800 raft_consensus.cc:1261] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Refusing update from remote peer a70d0bb08ec14c7ba294e396247795b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:14:51.344460 15133 consensus_queue.cc:1035] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:51.344604 15131 consensus_queue.cc:1035] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:14:51.346500 15136 mvcc.cc:205] Tried to move back new op lower bound from 7333241206142246912 to 7333241206029635584. Current Snapshot: MvccSnapshot[applied={T|T < 7333241206142246912}]
I20260925 13:14:51.349033 14658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:14:51.349135 14658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:14:51.349888 14658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Sending DeleteTablet for 3 replicas of tablet e5bcc8de9204491dadc394e3a72a220e
I20260925 13:14:51.350201 15050 tablet_service.cc:1514] Processing DeleteTablet for tablet e5bcc8de9204491dadc394e3a72a220e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36568
I20260925 13:14:51.350255 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet e5bcc8de9204491dadc394e3a72a220e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43008
I20260925 13:14:51.350351 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet e5bcc8de9204491dadc394e3a72a220e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49050
I20260925 13:14:51.350562 14855 tablet_replica.cc:330] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: stopping tablet replica
I20260925 13:14:51.350584 15146 tablet_replica.cc:330] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: stopping tablet replica
I20260925 13:14:51.350629 14855 raft_consensus.cc:2229] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:14:51.350688 14855 raft_consensus.cc:2258] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:14:51.350700 15146 raft_consensus.cc:2229] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:14:51.350806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14696
I20260925 13:14:51.350819 15146 raft_consensus.cc:2258] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:14:51.353026 15146 ts_tablet_manager.cc:1905] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:14:51.354739 15146 ts_tablet_manager.cc:1918] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:14:51.354820 15146 log.cc:1196] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/e5bcc8de9204491dadc394e3a72a220e
I20260925 13:14:51.355109 15146 ts_tablet_manager.cc:1939] T e5bcc8de9204491dadc394e3a72a220e P 8486fc0c411a4869a0f7303314260afa: Deleting consensus metadata
W20260925 13:14:51.357625 14642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42407 (error 108)
I20260925 13:14:51.357877 14642 catalog_manager.cc:4841] TS 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): tablet e5bcc8de9204491dadc394e3a72a220e (table test-table [id=74d6c3282d1f4b659f8820dc9847cdfe]) successfully deleted
I20260925 13:14:51.358013 15145 tablet_replica.cc:330] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: stopping tablet replica
I20260925 13:14:51.358093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14851
I20260925 13:14:51.358237 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.358124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42407 (local address 127.0.0.1:43022)
0925 13:14:51.358170 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.358204 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:14:51.358283 15145 raft_consensus.cc:2229] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:14:51.358386 15145 raft_consensus.cc:2258] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 13:14:51.358335 14642 catalog_manager.cc:4584] TS 88b4646f6e5e4d0baba74d04de59a243 (127.0.98.1:42407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5bcc8de9204491dadc394e3a72a220e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42407: connect: Connection refused (error 111)
I20260925 13:14:51.359169 15145 ts_tablet_manager.cc:1905] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:14:51.361147 15145 ts_tablet_manager.cc:1918] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:14:51.361212 15145 log.cc:1196] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/e5bcc8de9204491dadc394e3a72a220e
I20260925 13:14:51.361505 15145 ts_tablet_manager.cc:1939] T e5bcc8de9204491dadc394e3a72a220e P a70d0bb08ec14c7ba294e396247795b6: Deleting consensus metadata
I20260925 13:14:51.361951 14645 catalog_manager.cc:4841] TS a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): tablet e5bcc8de9204491dadc394e3a72a220e (table test-table [id=74d6c3282d1f4b659f8820dc9847cdfe]) successfully deleted
I20260925 13:14:51.364923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14987
I20260925 13:14:51.369856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 14625
I20260925 13:14:51.375198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--webserver_interface=127.0.98.62
--webserver_port=37701
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:51.457306 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:51.457553 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:51.457578 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:51.458909 15148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:14:51.458962 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:51.458977 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:51.458997 15148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:14:51.459061 15148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:14:51.460664 15148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:51.460939 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:51.461223 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:51.466192 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.466187 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.466434 15148 server_base.cc:1034] running on GCE node
W20260925 13:14:51.466216 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.466738 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:51.467017 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:51.468159 15148 hybrid_clock.cc:648] HybridClock initialized: now 1790342091468151 us; error 33 us; skew 500 ppm
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:51.470841 15148 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:14:51.471359 15148 webserver.cc:460] Webserver started at http://127.0.98.62:37701/ using document root <none> and password file <none>
I20260925 13:14:51.471617 15148 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:51.471689 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:51.473106 15148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.473853 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.474064 15148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.474174 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "9364433dedec48fda122bb8fb5449f28"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "edd2e4b70d9317f522417c943a579fa1"
server_key_iv: "ad0135cdca8c2cbb75224c266c310dc7"
server_key_version: "encryptionkey@0"
I20260925 13:14:51.474551 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:51.484200 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:51.486155 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:51.486310 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:51.490221 15148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35321
I20260925 13:14:51.490293 15215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35321 every 8 connection(s)
I20260925 13:14:51.490670 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:14:51.492280 15216 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:14:51.492969 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap starting.
I20260925 13:14:51.500344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15148
I20260925 13:14:51.500670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42407
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35321
--webserver_port=34809
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:51.501538 15216 log.cc:824] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:51.504001 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:14:51.504297 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28: Bootstrap complete.
I20260925 13:14:51.506673 15216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:51.506922 15216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364433dedec48fda122bb8fb5449f28, State: Initialized, Role: FOLLOWER
I20260925 13:14:51.507131 15216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:51.507243 15216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:14:51.507297 15216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:14:51.507347 15216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 2 FOLLOWER]: Advancing to term 3
I20260925 13:14:51.508364 15216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:51.508495 15216 leader_election.cc:302] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9364433dedec48fda122bb8fb5449f28; no voters:
I20260925 13:14:51.508706 15216 leader_election.cc:288] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 13:14:51.508746 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 13:14:51.508867 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [term 3 LEADER]: Becoming Leader. State: Replica: 9364433dedec48fda122bb8fb5449f28, State: Running, Role: LEADER
I20260925 13:14:51.508987 15216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:14:51.508955 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260925 13:14:51.509233 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9364433dedec48fda122bb8fb5449f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260925 13:14:51.509233 15222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9364433dedec48fda122bb8fb5449f28. Latest consensus state: current_term: 3 leader_uuid: "9364433dedec48fda122bb8fb5449f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9364433dedec48fda122bb8fb5449f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260925 13:14:51.509455 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:51.509469 15222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9364433dedec48fda122bb8fb5449f28 [sys.catalog]: This master's current role is: LEADER
I20260925 13:14:51.509853 15225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:14:51.510799 15225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:14:51.510957 15225 catalog_manager.cc:1259] Loaded cluster ID: 613390c31a2a476f8523f2a9cadd0528
I20260925 13:14:51.511003 15225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 13:14:51.511248 15225 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f44bffc49ad google::LogMessage::Fail() at ??:0
@ 0x7f44bffc788a google::LogMessage::SendToLog() at ??:0
@ 0x7f44bffc44de google::LogMessage::Flush() at ??:0
@ 0x7f44bffc5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f44c7ee50f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f44c7eee8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f44c1e920d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f44c1e87954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f44c2ceb6db start_thread at ??:0
@ 0x7f44beac261f clone at ??:0
W20260925 13:14:51.598634 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:51.598850 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:51.598875 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:51.600096 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:51.600138 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:51.600167 15218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:14:51.601634 15218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34809
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:51.601949 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:51.602205 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:51.607283 15252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.607295 15253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.607316 15255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.607631 15218 server_base.cc:1034] running on GCE node
I20260925 13:14:51.607870 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:51.608134 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:51.609305 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790342091609279 us; error 38 us; skew 500 ppm
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:51.612298 15218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:14:51.612826 15218 webserver.cc:460] Webserver started at http://127.0.98.1:34809/ using document root <none> and password file <none>
I20260925 13:14:51.613057 15218 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:51.613127 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:51.614984 15218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.615855 15262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.616114 15218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.616212 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "88b4646f6e5e4d0baba74d04de59a243"
format_stamp: "Formatted at 2026-09-25 13:14:49 on dist-test-slave-2d0s"
server_key: "1454e2af5aefdedd5f22693996457166"
server_key_iv: "d9f56c4a2f5b3dffa2749d97517908a2"
server_key_version: "encryptionkey@0"
I20260925 13:14:51.616708 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:51.640868 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:51.643543 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:51.643705 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:51.644014 15218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:51.644552 15269 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:14:51.646222 15218 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:14:51.646286 15218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:14:51.646320 15218 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:14:51.647222 15218 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:14:51.647266 15218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:14:51.647352 15269 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap starting.
I20260925 13:14:51.654281 15218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42407
I20260925 13:14:51.654479 15376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42407 every 8 connection(s)
I20260925 13:14:51.654755 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:14:51.656352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15218
I20260925 13:14:51.656721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37121
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35321
--webserver_port=33515
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:14:51.658550 15269 log.cc:824] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Log is configured to *not* fsync() on all Append() calls
I20260925 13:14:51.660684 15269 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:14:51.661015 15269 tablet_bootstrap.cc:492] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Bootstrap complete.
I20260925 13:14:51.662212 15269 ts_tablet_manager.cc:1397] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260925 13:14:51.664026 15269 raft_consensus.cc:348] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.664424 15269 raft_consensus.cc:729] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4646f6e5e4d0baba74d04de59a243, State: Initialized, Role: FOLLOWER
I20260925 13:14:51.664609 15269 consensus_queue.cc:260] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:51.664954 15269 ts_tablet_manager.cc:1428] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260925 13:14:51.716670 15379 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:14:51.655527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:53239)
0925 13:14:51.655740 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.655791 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:51.656427 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:51.716463 (+ 60036us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
W20260925 13:14:51.716948 15377 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35321 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35321: Client connection negotiation failed: client connection to 127.0.98.62:35321: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 13:14:51.717399 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.717266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:53451)
0925 13:14:51.717326 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.717373 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:14:51.717830 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.717733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:42473)
0925 13:14:51.717779 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.717807 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:14:51.717952 15377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:14:51.743486 15380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:51.743693 15380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:51.743717 15380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:51.744848 15380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:51.744882 15380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:51.744908 15380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:14:51.746348 15380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33515
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:51.746551 15380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:51.746727 15380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:51.751503 15387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.751549 15388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.751495 15390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.751801 15380 server_base.cc:1034] running on GCE node
I20260925 13:14:51.751987 15380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:51.752177 15380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:51.753312 15380 hybrid_clock.cc:648] HybridClock initialized: now 1790342091753304 us; error 31 us; skew 500 ppm
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:51.755897 15380 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:14:51.756309 15380 webserver.cc:460] Webserver started at http://127.0.98.2:33515/ using document root <none> and password file <none>
I20260925 13:14:51.756512 15380 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:51.756559 15380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:51.757884 15380 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.758476 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.758754 15380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.758836 15380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "8486fc0c411a4869a0f7303314260afa"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "f3740044c77e256b487a1f3e47fe5ffa"
server_key_iv: "2033d630dddef33a7d8634b55824f1ba"
server_key_version: "encryptionkey@0"
I20260925 13:14:51.759166 15380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:51.778447 15380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:51.780597 15380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:51.780706 15380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:51.780879 15380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:51.781195 15380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:51.781234 15380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.781257 15380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:51.781306 15380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.787112 15380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37121
I20260925 13:14:51.787205 15510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37121 every 8 connection(s)
I20260925 13:14:51.787540 15380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:14:51.788621 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.788235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:51053)
0925 13:14:51.788402 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.788473 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
W20260925 13:14:51.789016 15511 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35321 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35321: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
I20260925 13:14:51.789283 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.789190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:35841)
0925 13:14:51.789237 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.789261 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:14:51.789556 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.789473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:53835)
0925 13:14:51.789518 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.789539 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:14:51.789680 15511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:14:51.794649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15380
I20260925 13:14:51.794857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33897
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35321
--webserver_port=45945
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:14:51.874419 15514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:14:51.874641 15514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:14:51.874679 15514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:14:51.875829 15514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:14:51.875895 15514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:14:51.875938 15514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:14:51.877408 15514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45945
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:14:51.877689 15514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:14:51.877920 15514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:14:51.882844 15519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:14:51.882839 15520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.882856 15514 server_base.cc:1034] running on GCE node
W20260925 13:14:51.882836 15522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:14:51.883217 15514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:14:51.883396 15514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:14:51.884569 15514 hybrid_clock.cc:648] HybridClock initialized: now 1790342091884561 us; error 26 us; skew 500 ppm
Sep 25 13:14:51 dist-test-slave-2d0s krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:14:51.887089 15514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:14:51.887508 15514 webserver.cc:460] Webserver started at http://127.0.98.3:45945/ using document root <none> and password file <none>
I20260925 13:14:51.887733 15514 fs_manager.cc:359] Metadata directory not provided
I20260925 13:14:51.887777 15514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:14:51.889056 15514 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.889744 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.889959 15514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:14:51.890031 15514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "a70d0bb08ec14c7ba294e396247795b6"
format_stamp: "Formatted at 2026-09-25 13:14:50 on dist-test-slave-2d0s"
server_key: "6a319ed326cded926a2f7e1bdd615754"
server_key_iv: "bb8dc9c04f8d7dc9d2940ce72a5f74ea"
server_key_version: "encryptionkey@0"
I20260925 13:14:51.890377 15514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:14:51.901752 15514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:14:51.904047 15514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:14:51.904201 15514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:14:51.904448 15514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:14:51.904724 15514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:14:51.904757 15514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.904778 15514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:14:51.904791 15514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:14:51.910616 15514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33897
I20260925 13:14:51.910663 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33897 every 8 connection(s)
I20260925 13:14:51.911010 15514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:14:51.912190 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.911759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:45555)
0925 13:14:51.911984 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.912053 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":135,"threads_started":1}
W20260925 13:14:51.912637 15643 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35321 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35321: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
I20260925 13:14:51.912901 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.912817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:47975)
0925 13:14:51.912858 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.912884 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:14:51.913163 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:51.913089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:41461)
0925 13:14:51.913130 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:51.913150 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 13:14:51.913275 15643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:14:51.919821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15514
I20260925 13:14:52.718544 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:52.718330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:36343)
0925 13:14:52.718449 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:52.718488 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 13:14:52.790175 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:52.789968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:53599)
0925 13:14:52.790086 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:52.790125 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":50,"threads_started":1}
I20260925 13:14:52.913825 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:52.913584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:47007)
0925 13:14:52.913740 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:52.913774 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":53,"threads_started":1}
I20260925 13:14:53.001238 15650 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:53.001365 15650 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:53.001842 15650 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
Sep 25 13:14:53 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 13:14:53 dist-test-slave-2d0s krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 13:14:53.014490 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.002164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42973 (local address 127.0.98.2:37121)
0925 13:14:53.002303 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:53.002317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:53.002355 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:14:53.002960 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:53.002962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:53.003015 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:53.003169 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:53.004171 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.005590 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.007145 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.007785 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.007898 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:53.010861 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:53.010865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:53.010866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:53.010879 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:53.012520 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013025 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:53.013028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:53.013028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:53.013128 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013464 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:53.013466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:53.013466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:53.013798 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:53.013932 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:53.014230 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 13:14:53.014360 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260925 13:14:53.014508 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.001984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37121 (local address 127.0.98.1:42973)
0925 13:14:53.002066 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:53.002080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:53.002145 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:53.003240 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:53.003243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:53.003755 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:53.004040 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:53.004048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.005749 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.005751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:53.007048 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:53.007051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.007892 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.007893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:53.008034 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:53.008698 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:53.008702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:53.010694 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:53.012606 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:53.012612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:53.012614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:53.012915 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:53.013241 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:53.013243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:53.013381 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:53.013880 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:53.013889 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:53.014044 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 13:14:53.014161 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 13:14:53.014374 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":74}
I20260925 13:14:53.014516 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.002214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49001 (local address 127.0.98.3:33897)
0925 13:14:53.002359 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 13:14:53.002366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:14:53.002404 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:14:53.002952 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:14:53.002955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:14:53.003021 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:14:53.003169 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:14:53.004171 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.005698 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.007145 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.007785 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.007898 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:53.011167 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:14:53.011173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:14:53.011174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:14:53.011187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:14:53.013118 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013454 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:53.013457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:53.013458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:53.013554 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013778 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:14:53.013779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:14:53.013779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:14:53.014008 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:14:53.014102 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 13:14:53.014267 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 13:14:53.014399 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
I20260925 13:14:53.014760 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.002054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33897 (local address 127.0.98.1:49001)
0925 13:14:53.002334 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:53.002343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:14:53.002386 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:14:53.003240 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:14:53.003243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:14:53.003776 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:14:53.004040 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:53.004048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.005776 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.005776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:53.007048 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:14:53.007051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:14:53.007902 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:14:53.007903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:14:53.008034 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:14:53.008680 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:14:53.008691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:14:53.011047 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:14:53.013194 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:53.013195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:53.013361 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:53.013631 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:14:53.013632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:14:53.013632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:14:53.013713 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:14:53.014085 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:14:53.014086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:14:53.014132 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 13:14:53.014203 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:14:53.014437 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":150,"thread_start_us":64,"threads_started":1}
W20260925 13:14:53.015591 15266 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
W20260925 13:14:53.016397 15263 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
I20260925 13:14:53.016448 15263 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b4646f6e5e4d0baba74d04de59a243; no voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6
I20260925 13:14:53.016597 15650 raft_consensus.cc:2726] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:14:53.719260 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.719008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:34037)
0925 13:14:53.719149 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:53.719190 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260925 13:14:53.790928 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.790693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:36171)
0925 13:14:53.790827 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:53.790870 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260925 13:14:53.914443 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:53.914228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:37611)
0925 13:14:53.914365 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:53.914397 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260925 13:14:54.545881 15657 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:54.545984 15657 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:54.546150 15657 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
W20260925 13:14:54.546557 15266 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
W20260925 13:14:54.546653 15263 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
I20260925 13:14:54.546747 15263 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b4646f6e5e4d0baba74d04de59a243; no voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6
I20260925 13:14:54.546833 15657 raft_consensus.cc:2726] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:14:54.720016 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:54.719770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:47721)
0925 13:14:54.719933 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:54.719969 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260925 13:14:54.791567 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:54.791357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:40143)
0925 13:14:54.791483 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:54.791528 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260925 13:14:54.915177 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:54.914932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:42263)
0925 13:14:54.915068 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:54.915104 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260925 13:14:55.720709 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:55.720490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:40083)
0925 13:14:55.720612 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:55.720665 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260925 13:14:55.792205 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:55.792001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:45179)
0925 13:14:55.792136 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:55.792171 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 13:14:55.915803 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:55.915612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:54997)
0925 13:14:55.915733 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:55.915765 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260925 13:14:56.070866 15664 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:14:56.070971 15664 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:14:56.071125 15664 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
W20260925 13:14:56.071571 15266 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
W20260925 13:14:56.071658 15263 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
I20260925 13:14:56.071705 15263 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b4646f6e5e4d0baba74d04de59a243; no voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6
I20260925 13:14:56.071803 15664 raft_consensus.cc:2726] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:14:56.721398 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:56.721161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:45751)
0925 13:14:56.721285 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:56.721332 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260925 13:14:56.792963 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:56.792712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:44743)
0925 13:14:56.792856 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:56.792900 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260925 13:14:56.916462 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:56.916202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:52913)
0925 13:14:56.916361 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:56.916407 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260925 13:14:57.722141 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:57.721925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:54879)
0925 13:14:57.722055 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:57.722094 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260925 13:14:57.793706 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:57.793445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:35529)
0925 13:14:57.793575 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:57.793618 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260925 13:14:57.917178 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:57.916952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:49479)
0925 13:14:57.917100 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:57.917136 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260925 13:14:58.722878 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:58.722665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:39667)
0925 13:14:58.722799 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:58.722831 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 13:14:58.794281 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:58.794083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:42739)
0925 13:14:58.794204 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:58.794240 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260925 13:14:58.917977 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:58.917743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:54863)
0925 13:14:58.917893 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:58.917934 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260925 13:14:59.723812 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:59.723517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:41187)
0925 13:14:59.723708 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:59.723754 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260925 13:14:59.795028 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:59.794784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:37709)
0925 13:14:59.794924 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:59.794979 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260925 13:14:59.918838 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:14:59.918566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:56371)
0925 13:14:59.918728 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:14:59.918770 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260925 13:15:00.074040 15677 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:15:00.074134 15677 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:15:00.074281 15677 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
W20260925 13:15:00.074730 15263 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
W20260925 13:15:00.074805 15266 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
I20260925 13:15:00.074836 15266 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b4646f6e5e4d0baba74d04de59a243; no voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6
I20260925 13:15:00.074925 15677 raft_consensus.cc:2726] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:15:00.724617 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:00.724363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:42531)
0925 13:15:00.724523 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:00.724568 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260925 13:15:00.795725 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:00.795483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:44949)
0925 13:15:00.795636 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:00.795677 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260925 13:15:00.919487 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:00.919244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:48075)
0925 13:15:00.919388 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:00.919428 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260925 13:15:01.725373 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:01.725147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:49193)
0925 13:15:01.725283 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:01.725326 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260925 13:15:01.796531 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:01.796298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:48511)
0925 13:15:01.796445 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:01.796487 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260925 13:15:01.920404 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:01.920069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:54773)
0925 13:15:01.920312 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:01.920357 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":61,"threads_started":1}
I20260925 13:15:02.726197 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:02.725963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:51549)
0925 13:15:02.726119 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:02.726151 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260925 13:15:02.797322 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:02.797040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:52697)
0925 13:15:02.797197 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:02.797245 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260925 13:15:02.921149 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:02.920914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:40463)
0925 13:15:02.921057 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:02.921101 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260925 13:15:03.258919 15687 raft_consensus.cc:482] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:15:03.259035 15687 raft_consensus.cc:504] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8486fc0c411a4869a0f7303314260afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37121 } } peers { permanent_uuid: "a70d0bb08ec14c7ba294e396247795b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33897 } } peers { permanent_uuid: "88b4646f6e5e4d0baba74d04de59a243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42407 } }
I20260925 13:15:03.259248 15687 leader_election.cc:288] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121), a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897)
W20260925 13:15:03.259665 15263 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8486fc0c411a4869a0f7303314260afa (127.0.98.2:37121): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
W20260925 13:15:03.259737 15266 leader_election.cc:341] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a70d0bb08ec14c7ba294e396247795b6 (127.0.98.3:33897): Not found: Tablet not found: e5bcc8de9204491dadc394e3a72a220e
I20260925 13:15:03.259779 15266 leader_election.cc:302] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b4646f6e5e4d0baba74d04de59a243; no voters: 8486fc0c411a4869a0f7303314260afa, a70d0bb08ec14c7ba294e396247795b6
I20260925 13:15:03.259850 15687 raft_consensus.cc:2726] T e5bcc8de9204491dadc394e3a72a220e P 88b4646f6e5e4d0baba74d04de59a243 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:15:03.726847 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:03.726625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:43445)
0925 13:15:03.726757 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:03.726798 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260925 13:15:03.798087 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:03.797861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:39979)
0925 13:15:03.797983 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:03.798022 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260925 13:15:03.921887 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:03.921618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:56619)
0925 13:15:03.921807 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:03.921842 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":124,"threads_started":1}
I20260925 13:15:04.727628 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:04.727371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:37011)
0925 13:15:04.727530 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:04.727578 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260925 13:15:04.798738 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:04.798524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:35035)
0925 13:15:04.798646 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:04.798690 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260925 13:15:04.922588 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:04.922348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:34179)
0925 13:15:04.922488 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:04.922541 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260925 13:15:05.728363 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:05.728109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:39271)
0925 13:15:05.728272 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:05.728313 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
I20260925 13:15:05.799458 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:05.799227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:44455)
0925 13:15:05.799364 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:05.799407 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260925 13:15:05.923283 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:05.923040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:35925)
0925 13:15:05.923198 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:05.923237 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260925 13:15:06.729197 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:06.728975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:60305)
0925 13:15:06.729122 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:06.729150 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260925 13:15:06.800184 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:06.799959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:51527)
0925 13:15:06.800103 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:06.800140 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260925 13:15:06.924015 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:06.923782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:59435)
0925 13:15:06.923934 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:06.923971 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260925 13:15:06.929246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 15218
I20260925 13:15:06.934101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 15380
I20260925 13:15:06.938380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 15514
2026-09-25T13:15:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18454 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:15:06 dist-test-slave-2d0s krb5kdc[15702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:15:06 dist-test-slave-2d0s krb5kdc[15702](info): set up 2 sockets
Sep 25 13:15:06 dist-test-slave-2d0s krb5kdc[15702](info): commencing operation
krb5kdc: starting...
W20260925 13:15:07.969877 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:15:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:15:08.093169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:08.174211 15715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:08.174481 15715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:08.174520 15715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:08.175714 15715 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 13:15:08.175794 15715 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 13:15:08.175817 15715 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 13:15:08.175836 15715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:15:08.175868 15715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:08.175891 15715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:08.175920 15715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:15:08.175948 15715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:15:08.177475 15715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:08.177809 15715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:08.178046 15715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:08.182909 15720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.182950 15721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.182909 15723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:08.183043 15715 server_base.cc:1034] running on GCE node
I20260925 13:15:08.183403 15715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:08.183737 15715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:08.184883 15715 hybrid_clock.cc:648] HybridClock initialized: now 1790342108184856 us; error 43 us; skew 500 ppm
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:08.187403 15715 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:15:08.187880 15715 webserver.cc:460] Webserver started at http://127.0.98.62:33629/ using document root <none> and password file <none>
I20260925 13:15:08.188100 15715 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:08.188174 15715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:08.188328 15715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:15:08.189394 15715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "9ac284ee121f4d9592df3dbaa491c0c0"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "9e1b8c762cedc09bac8764c630789dc9"
server_key_iv: "ce23ab187eb9483b97584c52e6b3a3e6"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.189800 15715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ac284ee121f4d9592df3dbaa491c0c0"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "9e1b8c762cedc09bac8764c630789dc9"
server_key_iv: "ce23ab187eb9483b97584c52e6b3a3e6"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.191167 15715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:15:08.191869 15730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.192049 15715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:15:08.192155 15715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "9ac284ee121f4d9592df3dbaa491c0c0"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "9e1b8c762cedc09bac8764c630789dc9"
server_key_iv: "ce23ab187eb9483b97584c52e6b3a3e6"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.192240 15715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:08.200412 15715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:08.202844 15715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:08.203013 15715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:08.206888 15715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46859
I20260925 13:15:08.206921 15782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46859 every 8 connection(s)
I20260925 13:15:08.207285 15715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:15:08.208140 15783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:08.208238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15715
I20260925 13:15:08.208341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:15:08.208544 392 external_mini_cluster.cc:1428] Setting key b431a65c06c7eab186ad4eec1a52b7e3
I20260925 13:15:08.211851 15783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap starting.
I20260925 13:15:08.212564 15783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:08.212883 15783 log.cc:824] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:08.213634 15783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: No bootstrap required, opened a new log
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:08.216354 15783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:08.216562 15783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:08.216604 15783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac284ee121f4d9592df3dbaa491c0c0, State: Initialized, Role: FOLLOWER
I20260925 13:15:08.216766 15783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:08.216876 15783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:15:08.216920 15783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:15:08.217000 15783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:08.217903 15783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:08.218019 15783 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac284ee121f4d9592df3dbaa491c0c0; no voters:
I20260925 13:15:08.218184 15783 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:15:08.218281 15788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:15:08.218456 15788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 LEADER]: Becoming Leader. State: Replica: 9ac284ee121f4d9592df3dbaa491c0c0, State: Running, Role: LEADER
I20260925 13:15:08.218489 15783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:15:08.218575 15788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:08.218972 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260925 13:15:08.218964 15788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac284ee121f4d9592df3dbaa491c0c0. Latest consensus state: current_term: 1 leader_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260925 13:15:08.219151 15788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:15:08.219144 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:15:08.219404 15797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:15:08.219957 15797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:15:08.220816 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.209066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.0.98.62:46859)
0925 13:15:08.209324 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.209334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.209365 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:15:08.209956 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.209966 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.210035 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.210202 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:08.210902 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.212308 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.213194 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.213752 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.213886 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.216467 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:08.216471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:08.216471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:08.216483 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:08.218621 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.219053 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.219056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.219057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.219211 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.219435 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.219436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.219436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.219767 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:08.219963 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.220549 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.220673 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":71,"threads_started":1}
I20260925 13:15:08.222864 15797 catalog_manager.cc:1347] Generated new cluster ID: 1f7c00e498e1477e94d5d52959d42ea6
I20260925 13:15:08.222921 15797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:15:08.236708 15797 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:15:08.237504 15797 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:15:08.248682 15797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Generated new TSK 0
I20260925 13:15:08.248924 15797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:15:08.286243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:08.372850 15809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:08.373122 15809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:08.373167 15809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:08.374430 15809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:08.374511 15809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:08.374559 15809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:15:08.376071 15809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:08.376344 15809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:08.376612 15809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:08.381896 15814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.381904 15815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.381907 15817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:08.382063 15809 server_base.cc:1034] running on GCE node
I20260925 13:15:08.382371 15809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:08.382638 15809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:08.383785 15809 hybrid_clock.cc:648] HybridClock initialized: now 1790342108383779 us; error 34 us; skew 500 ppm
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:08.386407 15809 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:15:08.386806 15809 webserver.cc:460] Webserver started at http://127.0.98.1:46075/ using document root <none> and password file <none>
I20260925 13:15:08.387059 15809 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:08.387131 15809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:08.387286 15809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:15:08.388414 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad631a2adb5a430a9f467cfc128a703b"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "b1e6efdeec86446f721fd85b28fdc85c"
server_key_iv: "be2c0889181ccf06e4fced79873975da"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.388777 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad631a2adb5a430a9f467cfc128a703b"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "b1e6efdeec86446f721fd85b28fdc85c"
server_key_iv: "be2c0889181ccf06e4fced79873975da"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.390180 15809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 13:15:08.390936 15824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.391111 15809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:15:08.391227 15809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "ad631a2adb5a430a9f467cfc128a703b"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "b1e6efdeec86446f721fd85b28fdc85c"
server_key_iv: "be2c0889181ccf06e4fced79873975da"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.391304 15809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:08.404683 15809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:08.406817 15809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:08.406980 15809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:08.407229 15809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:08.407538 15809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:15:08.407593 15809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.407639 15809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:15:08.407670 15809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.413652 15809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36837
I20260925 13:15:08.413755 15937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36837 every 8 connection(s)
I20260925 13:15:08.414148 15809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:08.421991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15809
I20260925 13:15:08.422106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:15:08.422317 392 external_mini_cluster.cc:1428] Setting key 9bccc5f4c6ac6e455835f27102d7e276
I20260925 13:15:08.423962 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.414971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41961 (local address 127.0.98.62:46859)
0925 13:15:08.415064 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.415066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.415227 (+ 161us) server_negotiation.cc:407] Connection header received
0925 13:15:08.415704 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.415705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.415716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.415776 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:08.416754 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.417263 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.418282 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.418597 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.418652 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.421059 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:08.421061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:08.421061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:08.421067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:08.422528 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.422923 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.422925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.422925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.422995 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.423328 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.423328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.423329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.423433 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:08.423486 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.423829 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.423899 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 13:15:08.424288 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.414920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:41961)
0925 13:15:08.415079 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.415117 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.415615 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.415847 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.415851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.416448 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:08.416663 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.416668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.417346 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.417346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.418204 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.418205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.418703 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.418704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.418829 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.419432 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:08.419441 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:08.420972 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:08.422592 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:08.422600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:08.422603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:08.422836 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:08.423078 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:08.423079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:08.423079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:08.423253 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:08.423502 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:08.423508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:08.423691 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.423810 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.423993 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260925 13:15:08.424898 15938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260925 13:15:08.425011 15938 heartbeater.cc:464] Registering TS with master...
I20260925 13:15:08.425220 15938 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260925 13:15:08.425839 15747 ts_manager.cc:194] Registered new tserver with Master: ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
I20260925 13:15:08.426460 15747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41961
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:15:08.448403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:08.531919 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:08.532150 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:08.532174 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:08.533420 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:08.533458 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:08.533484 15943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:15:08.534926 15943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:08.535125 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:08.535311 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:08.540428 15951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.540436 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:08.540521 15943 server_base.cc:1034] running on GCE node
W20260925 13:15:08.540428 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:08.540853 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:08.541124 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:08.542266 15943 hybrid_clock.cc:648] HybridClock initialized: now 1790342108542249 us; error 30 us; skew 500 ppm
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:08.545028 15943 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:15:08.545516 15943 webserver.cc:460] Webserver started at http://127.0.98.2:40457/ using document root <none> and password file <none>
I20260925 13:15:08.545835 15943 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:08.545883 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:08.546027 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:15:08.547149 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "b49845143cec411d9e6279854bcd96e7"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "68dd0d625914d12561d52a6c625727fa"
server_key_iv: "7909cb55c1310c46870a7231f989cbb1"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.547519 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b49845143cec411d9e6279854bcd96e7"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "68dd0d625914d12561d52a6c625727fa"
server_key_iv: "7909cb55c1310c46870a7231f989cbb1"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.548830 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:15:08.549607 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.549796 15943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:15:08.549890 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "b49845143cec411d9e6279854bcd96e7"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "68dd0d625914d12561d52a6c625727fa"
server_key_iv: "7909cb55c1310c46870a7231f989cbb1"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.549968 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:08.588290 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:08.590489 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:08.590674 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:08.590929 15943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:08.591264 15943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:15:08.591297 15943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.591343 15943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:15:08.591372 15943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.597719 15943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37207
I20260925 13:15:08.597745 16071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37207 every 8 connection(s)
I20260925 13:15:08.598097 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:15:08.604174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 15943
I20260925 13:15:08.604276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:15:08.604485 392 external_mini_cluster.cc:1428] Setting key 42f72748733efb0f4bff0046487d0dd0
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:08.608461 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.598810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48709 (local address 127.0.98.62:46859)
0925 13:15:08.598855 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.598857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.599120 (+ 263us) server_negotiation.cc:407] Connection header received
0925 13:15:08.599678 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.599680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.599693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.599763 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:08.600784 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.601300 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.602398 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.602604 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.602692 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.605197 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:08.605198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:08.605198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:08.605207 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:08.606913 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.607369 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.607370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.607371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.607442 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.607703 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.607710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.607711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.607872 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:08.607943 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.608218 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.608315 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:15:08.608494 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.598779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:48709)
0925 13:15:08.598971 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.599028 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.599566 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.599840 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.599845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.600470 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:08.600675 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.600681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.601410 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.601411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.602308 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.602311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.602702 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.602704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.602845 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.603424 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:08.603433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:08.605094 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:08.607010 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:08.607017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:08.607019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:08.607288 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:08.607517 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:08.607518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:08.607518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:08.607639 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:08.607940 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:08.607945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:08.608080 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.608173 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.608314 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260925 13:15:08.609189 16072 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260925 13:15:08.609313 16072 heartbeater.cc:464] Registering TS with master...
I20260925 13:15:08.609520 16072 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260925 13:15:08.610091 15747 ts_manager.cc:194] Registered new tserver with Master: b49845143cec411d9e6279854bcd96e7 (127.0.98.2:37207)
I20260925 13:15:08.610445 15747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48709
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:15:08.641080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:08.724260 16077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:08.724491 16077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:08.724532 16077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:08.725750 16077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:08.725805 16077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:08.725849 16077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:15:08.727311 16077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:08.727576 16077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:08.727807 16077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:08.732995 16083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.733024 16085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:08.732995 16082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:08.733261 16077 server_base.cc:1034] running on GCE node
I20260925 13:15:08.733454 16077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:08.733654 16077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:08.734864 16077 hybrid_clock.cc:648] HybridClock initialized: now 1790342108734845 us; error 34 us; skew 500 ppm
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:08.737351 16077 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:15:08.737857 16077 webserver.cc:460] Webserver started at http://127.0.98.3:40671/ using document root <none> and password file <none>
I20260925 13:15:08.738091 16077 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:08.738158 16077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:08.738310 16077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:15:08.739351 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf98ce4c32594601a3ef73f751033caf"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "0a6b34668fa774ccd263ea84708cf963"
server_key_iv: "c7d8bef07d6039bd3e66432a1d0a947b"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.739714 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf98ce4c32594601a3ef73f751033caf"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "0a6b34668fa774ccd263ea84708cf963"
server_key_iv: "c7d8bef07d6039bd3e66432a1d0a947b"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.740998 16077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:15:08.741707 16092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.741922 16077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:15:08.742019 16077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "bf98ce4c32594601a3ef73f751033caf"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "0a6b34668fa774ccd263ea84708cf963"
server_key_iv: "c7d8bef07d6039bd3e66432a1d0a947b"
server_key_version: "encryptionkey@0"
I20260925 13:15:08.742105 16077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:08.757171 16077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:08.759176 16077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:08.759343 16077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:08.759573 16077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:08.759928 16077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:15:08.759990 16077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.760037 16077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:15:08.760077 16077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:08.765946 16077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45607
I20260925 13:15:08.765998 16205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45607 every 8 connection(s)
I20260925 13:15:08.766312 16077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:15:08 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:08.775624 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.767072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46407 (local address 127.0.98.62:46859)
0925 13:15:08.767124 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.767125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.767365 (+ 240us) server_negotiation.cc:407] Connection header received
0925 13:15:08.767900 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.767901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.767914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.767967 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:08.768953 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.769462 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.770570 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.770778 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.770853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.773016 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:08.773017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:08.773017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:08.773026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:08.774235 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.774647 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.774648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.774648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.774708 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.774973 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.774974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.774975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.775086 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:08.775159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.775451 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.775529 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:15:08.775825 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.767042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:46407)
0925 13:15:08.767211 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.767261 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.767791 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.768054 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.768058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.768649 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:08.768846 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.768851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.769561 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.769562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.770467 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.770469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.770862 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.770863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.771003 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.771579 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:08.771587 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:08.772918 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:08.774334 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:08.774341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:08.774343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:08.774557 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:08.774790 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:08.774791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:08.774791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:08.774897 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:08.775162 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:08.775169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:08.775316 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.775418 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.775578 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260925 13:15:08.776072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16077
I20260925 13:15:08.776173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:15:08.776340 16206 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260925 13:15:08.776407 392 external_mini_cluster.cc:1428] Setting key 20411e4ca58d5ee6f849c0ae5aa6d349
I20260925 13:15:08.776435 16206 heartbeater.cc:464] Registering TS with master...
I20260925 13:15:08.776623 16206 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260925 13:15:08.777063 15747 ts_manager.cc:194] Registered new tserver with Master: bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607)
I20260925 13:15:08.777395 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:15:08.777590 15747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46407
I20260925 13:15:08.784211 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.778864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.0.98.62:46859)
0925 13:15:08.778903 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.778905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.779033 (+ 128us) server_negotiation.cc:407] Connection header received
0925 13:15:08.779054 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.779055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.779063 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.779105 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:08.779580 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.780017 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.780737 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.780910 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.780957 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.782115 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:08.782116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:08.782116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:08.782122 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:08.783297 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.783596 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.783597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.783597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.783644 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:08.783805 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:08.783806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:08.783806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:08.783933 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:08.784005 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.784092 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.784147 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:15:08.785828 15747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:15:08.787438 15747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:15:08.793843 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.789819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.0.1:37508)
0925 13:15:08.790219 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.790253 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.790339 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.790564 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.790568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.790571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:08.790645 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.790647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.791608 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.791608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.792627 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.792629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.793451 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.793452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.793584 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.793603 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.793689 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.793763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260925 13:15:08.793864 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.789928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.98.1:36837)
0925 13:15:08.790135 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.790141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.790312 (+ 171us) server_negotiation.cc:407] Connection header received
0925 13:15:08.790447 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.790450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.790469 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.790553 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:08.790759 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.791518 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.792759 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.793363 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.793437 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.793496 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.793731 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.793779 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260925 13:15:08.794652 15872 tablet_service.cc:1467] Processing CreateTablet for tablet c4e086b5185f463fa7e75f6876197fef (DEFAULT_TABLE table=test-table [id=d245b51b9f9142bc9e439e459dd74e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:15:08.795095 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e086b5185f463fa7e75f6876197fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:08.795357 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.789819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45607 (local address 127.0.0.1:59436)
0925 13:15:08.790201 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.790242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.790319 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.790776 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.790777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.790779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:08.790827 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.790829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.793141 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.793141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.794103 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.794105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.795103 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.795104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.795173 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.795177 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.795244 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.795288 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"spinlock_wait_cycles":384,"thread_start_us":90,"threads_started":1}
I20260925 13:15:08.795425 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.790001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.98.3:45607)
0925 13:15:08.790178 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.790182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.790538 (+ 356us) server_negotiation.cc:407] Connection header received
0925 13:15:08.790585 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.790588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.790604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.790675 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:08.790904 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.792027 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.794186 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.794935 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.795007 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.795065 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.795283 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.795333 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260925 13:15:08.796119 16140 tablet_service.cc:1467] Processing CreateTablet for tablet c4e086b5185f463fa7e75f6876197fef (DEFAULT_TABLE table=test-table [id=d245b51b9f9142bc9e439e459dd74e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:15:08.796474 16140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e086b5185f463fa7e75f6876197fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:08.798269 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.790013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37207 (local address 127.0.0.1:51058)
0925 13:15:08.790421 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.790429 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.790480 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.790813 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.790814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.790815 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:08.790868 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.790870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.792137 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.792137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.793071 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.793073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.798023 (+ 4950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.798024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.798099 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.798103 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.798157 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.798200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":74,"threads_started":1}
I20260925 13:15:08.798317 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.790155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.98.2:37207)
0925 13:15:08.790304 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.790309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.790663 (+ 354us) server_negotiation.cc:407] Connection header received
0925 13:15:08.790709 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.790712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.790725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.790801 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:08.790923 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.792023 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.793154 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.797925 (+ 4771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.798013 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.798076 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.798173 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.798219 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260925 13:15:08.798771 16223 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Bootstrap starting.
I20260925 13:15:08.799039 16006 tablet_service.cc:1467] Processing CreateTablet for tablet c4e086b5185f463fa7e75f6876197fef (DEFAULT_TABLE table=test-table [id=d245b51b9f9142bc9e439e459dd74e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:15:08.799340 16006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e086b5185f463fa7e75f6876197fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:08.799459 16223 tablet_bootstrap.cc:654] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:08.799916 16224 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Bootstrap starting.
I20260925 13:15:08.800100 16223 log.cc:824] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:08.800576 16224 tablet_bootstrap.cc:654] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:08.800844 16223 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: No bootstrap required, opened a new log
I20260925 13:15:08.800935 16223 ts_tablet_manager.cc:1397] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:15:08.801245 16224 log.cc:824] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:08.802430 16224 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: No bootstrap required, opened a new log
I20260925 13:15:08.802501 16224 ts_tablet_manager.cc:1397] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:15:08.802723 16227 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Bootstrap starting.
I20260925 13:15:08.802973 16223 raft_consensus.cc:348] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.803182 16223 raft_consensus.cc:374] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:08.803217 16223 raft_consensus.cc:729] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad631a2adb5a430a9f467cfc128a703b, State: Initialized, Role: FOLLOWER
I20260925 13:15:08.803359 16223 consensus_queue.cc:260] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.803422 16227 tablet_bootstrap.cc:654] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:08.803663 16223 ts_tablet_manager.cc:1428] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:15:08.803830 15938 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260925 13:15:08.803992 16227 log.cc:824] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:08.804388 16224 raft_consensus.cc:348] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.804556 16224 raft_consensus.cc:374] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:08.804600 16224 raft_consensus.cc:729] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf98ce4c32594601a3ef73f751033caf, State: Initialized, Role: FOLLOWER
I20260925 13:15:08.804713 16227 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: No bootstrap required, opened a new log
I20260925 13:15:08.804718 16224 consensus_queue.cc:260] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.804781 16227 ts_tablet_manager.cc:1397] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:15:08.805003 16224 ts_tablet_manager.cc:1428] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:15:08.805176 16206 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260925 13:15:08.806655 16227 raft_consensus.cc:348] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.806804 16227 raft_consensus.cc:374] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:08.806849 16227 raft_consensus.cc:729] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49845143cec411d9e6279854bcd96e7, State: Initialized, Role: FOLLOWER
I20260925 13:15:08.806957 16227 consensus_queue.cc:260] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.807250 16227 ts_tablet_manager.cc:1428] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:15:08.807273 16072 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
W20260925 13:15:08.848713 16073 tablet.cc:2369] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:15:08.863507 16231 raft_consensus.cc:482] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:15:08.863682 16231 raft_consensus.cc:504] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.864159 16231 leader_election.cc:288] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607), ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
I20260925 13:15:08.867139 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.864455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50555 (local address 127.0.98.3:45607)
0925 13:15:08.864500 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.864503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.864625 (+ 122us) server_negotiation.cc:407] Connection header received
0925 13:15:08.864657 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.864658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.864670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.864735 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:08.864900 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.865366 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.866390 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.866773 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.866829 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.866837 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.867024 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.867068 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:15:08.867151 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.864309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.98.2:53227)
0925 13:15:08.864372 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.864386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.864434 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.864575 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.864576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.864578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:08.864676 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.864678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.865392 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.865393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.866290 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.866293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.866895 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.866896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.866963 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.866966 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.867037 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.867073 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":44}
I20260925 13:15:08.867170 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.864395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53227 (local address 127.0.98.1:36837)
0925 13:15:08.864452 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.864454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.864463 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:15:08.864493 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.864494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.864508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.864568 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:08.864759 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.865305 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.866389 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.866785 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.866877 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.866887 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.867061 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.867107 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:15:08.867170 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.864312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45607 (local address 127.0.98.2:50555)
0925 13:15:08.864547 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.864556 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:08.864604 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:08.864755 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:08.864756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:08.864757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:08.864826 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.864828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.865459 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.865460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.866291 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:08.866293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.866862 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.866863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:08.866947 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.866952 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:15:08.866994 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 13:15:08.867056 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":69,"thread_start_us":113,"threads_started":1}
I20260925 13:15:08.867532 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e086b5185f463fa7e75f6876197fef" candidate_uuid: "b49845143cec411d9e6279854bcd96e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad631a2adb5a430a9f467cfc128a703b" is_pre_election: true
I20260925 13:15:08.867717 15892 raft_consensus.cc:2454] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b49845143cec411d9e6279854bcd96e7 in term 0.
I20260925 13:15:08.867715 16160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e086b5185f463fa7e75f6876197fef" candidate_uuid: "b49845143cec411d9e6279854bcd96e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf98ce4c32594601a3ef73f751033caf" is_pre_election: true
I20260925 13:15:08.867899 16160 raft_consensus.cc:2454] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b49845143cec411d9e6279854bcd96e7 in term 0.
I20260925 13:15:08.867988 15962 leader_election.cc:302] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad631a2adb5a430a9f467cfc128a703b, b49845143cec411d9e6279854bcd96e7; no voters:
I20260925 13:15:08.868127 16231 raft_consensus.cc:2781] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:15:08.868222 16231 raft_consensus.cc:482] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:15:08.868255 16231 raft_consensus.cc:3037] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:08.868985 16231 raft_consensus.cc:504] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.869138 16231 leader_election.cc:288] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [CANDIDATE]: Term 1 election: Requested vote from peers bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607), ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
I20260925 13:15:08.869294 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e086b5185f463fa7e75f6876197fef" candidate_uuid: "b49845143cec411d9e6279854bcd96e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad631a2adb5a430a9f467cfc128a703b"
I20260925 13:15:08.869310 16160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e086b5185f463fa7e75f6876197fef" candidate_uuid: "b49845143cec411d9e6279854bcd96e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf98ce4c32594601a3ef73f751033caf"
I20260925 13:15:08.869386 15892 raft_consensus.cc:3037] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:08.869381 16160 raft_consensus.cc:3037] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:08.870185 15892 raft_consensus.cc:2454] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b49845143cec411d9e6279854bcd96e7 in term 1.
I20260925 13:15:08.870193 16160 raft_consensus.cc:2454] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b49845143cec411d9e6279854bcd96e7 in term 1.
I20260925 13:15:08.870365 15959 leader_election.cc:302] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b49845143cec411d9e6279854bcd96e7, bf98ce4c32594601a3ef73f751033caf; no voters:
I20260925 13:15:08.870467 16231 raft_consensus.cc:2781] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:15:08.870628 16231 raft_consensus.cc:686] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 1 LEADER]: Becoming Leader. State: Replica: b49845143cec411d9e6279854bcd96e7, State: Running, Role: LEADER
I20260925 13:15:08.870745 16231 consensus_queue.cc:237] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:08.871623 15747 catalog_manager.cc:5495] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b49845143cec411d9e6279854bcd96e7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b49845143cec411d9e6279854bcd96e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260925 13:15:08.887319 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.884811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.0.98.2:37207)
0925 13:15:08.884876 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:08.884878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:08.884888 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:15:08.884933 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:08.884934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:08.884954 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:08.885032 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:15:08.885168 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.885648 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.886504 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:08.886703 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:08.886781 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:08.886955 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:15:08.887062 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:15:08.887151 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:08.887192 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 13:15:08.887213 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:15:08.889168 15892 raft_consensus.cc:1261] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Refusing update from remote peer b49845143cec411d9e6279854bcd96e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:15:08.889235 16160 raft_consensus.cc:1261] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Refusing update from remote peer b49845143cec411d9e6279854bcd96e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:15:08.889578 16231 consensus_queue.cc:1035] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:15:08.889744 16233 consensus_queue.cc:1035] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:15:08.892469 16238 mvcc.cc:205] Tried to move back new op lower bound from 7333241278007189504 to 7333241277935198208. Current Snapshot: MvccSnapshot[applied={T|T < 7333241278007189504}]
I20260925 13:15:08.892769 16237 mvcc.cc:205] Tried to move back new op lower bound from 7333241278007189504 to 7333241277935198208. Current Snapshot: MvccSnapshot[applied={T|T < 7333241278007189504}]
I20260925 13:15:08.894244 15746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:15:08.894375 15746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:15:08.895117 15746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Sending DeleteTablet for 3 replicas of tablet c4e086b5185f463fa7e75f6876197fef
I20260925 13:15:08.895479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 15809
I20260925 13:15:08.895463 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e086b5185f463fa7e75f6876197fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37508
I20260925 13:15:08.895512 16140 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e086b5185f463fa7e75f6876197fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59436
I20260925 13:15:08.895692 16006 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e086b5185f463fa7e75f6876197fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51058
I20260925 13:15:08.896997 16246 tablet_replica.cc:330] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: stopping tablet replica
I20260925 13:15:08.897099 16246 raft_consensus.cc:2229] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:15:08.897192 16246 raft_consensus.cc:2258] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:15:08.897342 16245 tablet_replica.cc:330] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: stopping tablet replica
I20260925 13:15:08.897429 16245 raft_consensus.cc:2229] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:15:08.897526 16245 raft_consensus.cc:2258] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:15:08.898396 16245 ts_tablet_manager.cc:1905] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:15:08.899389 16246 ts_tablet_manager.cc:1905] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:15:08.900385 16245 ts_tablet_manager.cc:1918] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:15:08.900463 16245 log.cc:1196] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/c4e086b5185f463fa7e75f6876197fef
I20260925 13:15:08.900743 16245 ts_tablet_manager.cc:1939] T c4e086b5185f463fa7e75f6876197fef P bf98ce4c32594601a3ef73f751033caf: Deleting consensus metadata
I20260925 13:15:08.901751 16246 ts_tablet_manager.cc:1918] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:15:08.901826 16246 log.cc:1196] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/c4e086b5185f463fa7e75f6876197fef
I20260925 13:15:08.902051 15731 catalog_manager.cc:4841] TS bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607): tablet c4e086b5185f463fa7e75f6876197fef (table test-table [id=d245b51b9f9142bc9e439e459dd74e30]) successfully deleted
I20260925 13:15:08.902187 16246 ts_tablet_manager.cc:1939] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Deleting consensus metadata
W20260925 13:15:08.902441 15734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36837 (error 108)
I20260925 13:15:08.902608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 15943
I20260925 13:15:08.903967 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.903684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.0.1:37520)
0925 13:15:08.903871 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.903923 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260925 13:15:08.904115 15734 catalog_manager.cc:4584] TS ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4e086b5185f463fa7e75f6876197fef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36837: connect: Connection refused (error 111)
I20260925 13:15:08.908157 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:08.908021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37207 (local address 127.0.0.1:51084)
0925 13:15:08.908083 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:08.908111 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 13:15:08.908202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 16077
I20260925 13:15:08.912858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 15715
I20260925 13:15:28.918547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--webserver_interface=127.0.98.62
--webserver_port=33629
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:29.001441 16248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:29.001695 16248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:29.001719 16248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:29.002931 16248 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 13:15:29.002975 16248 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 13:15:29.002988 16248 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 13:15:29.003000 16248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:15:29.003013 16248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:29.003026 16248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:29.003075 16248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:15:29.003093 16248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:15:29.004616 16248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:29.004837 16248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:29.005023 16248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:29.014110 16255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:29.014110 16257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:29.014113 16254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:29.014369 16248 server_base.cc:1034] running on GCE node
I20260925 13:15:29.014655 16248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:29.015027 16248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:29.016188 16248 hybrid_clock.cc:648] HybridClock initialized: now 1790342129016164 us; error 38 us; skew 500 ppm
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:29.022965 16248 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:15:29.023578 16248 webserver.cc:460] Webserver started at http://127.0.98.62:33629/ using document root <none> and password file <none>
I20260925 13:15:29.023847 16248 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:29.023939 16248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:29.025424 16248 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:15:29.026216 16264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.026448 16248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:15:29.026563 16248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "9ac284ee121f4d9592df3dbaa491c0c0"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "9e1b8c762cedc09bac8764c630789dc9"
server_key_iv: "ce23ab187eb9483b97584c52e6b3a3e6"
server_key_version: "encryptionkey@0"
I20260925 13:15:29.027014 16248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:29.035015 16248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:29.037417 16248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:29.037590 16248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:29.041862 16248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46859
I20260925 13:15:29.041898 16316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46859 every 8 connection(s)
I20260925 13:15:29.042330 16248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:15:29.043785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16248
I20260925 13:15:29.043843 16317 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:15:29.044082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36837
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46859
--webserver_port=46075
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:15:29.044623 16317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap starting.
I20260925 13:15:29.053158 16317 log.cc:824] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:29.055224 16317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:15:29.055531 16317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap complete.
I20260925 13:15:29.057937 16317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:29.058238 16317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac284ee121f4d9592df3dbaa491c0c0, State: Initialized, Role: FOLLOWER
I20260925 13:15:29.058431 16317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:29.058508 16317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:15:29.058566 16317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:15:29.058614 16317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:15:29.059719 16317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:29.059842 16317 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac284ee121f4d9592df3dbaa491c0c0; no voters:
I20260925 13:15:29.060055 16317 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:15:29.060091 16321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:15:29.060212 16321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 LEADER]: Becoming Leader. State: Replica: 9ac284ee121f4d9592df3dbaa491c0c0, State: Running, Role: LEADER
I20260925 13:15:29.060338 16317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:15:29.060304 16321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:29.060617 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac284ee121f4d9592df3dbaa491c0c0. Latest consensus state: current_term: 2 leader_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260925 13:15:29.060684 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:15:29.060854 16322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260925 13:15:29.060940 16322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:15:29.061038 16325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:15:29.061926 16325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:15:29.062100 16325 catalog_manager.cc:1259] Loaded cluster ID: 1f7c00e498e1477e94d5d52959d42ea6
I20260925 13:15:29.062304 16325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:15:29.063512 16325 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:15:29.063699 16325 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Loaded TSK: 0
I20260925 13:15:29.073952 16325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Generated new TSK 1
I20260925 13:15:29.074060 16325 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:15:29.129710 16319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:29.129956 16319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:29.129997 16319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:29.131289 16319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:29.131352 16319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:29.131400 16319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:15:29.132901 16319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46075
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:29.133185 16319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:29.133428 16319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:29.138548 16344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:29.138577 16345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:29.138744 16319 server_base.cc:1034] running on GCE node
W20260925 13:15:29.138577 16347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:29.139051 16319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:29.139343 16319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:29.140586 16319 hybrid_clock.cc:648] HybridClock initialized: now 1790342129140573 us; error 51 us; skew 500 ppm
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:29.143649 16319 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:15:29.144141 16319 webserver.cc:460] Webserver started at http://127.0.98.1:46075/ using document root <none> and password file <none>
I20260925 13:15:29.144394 16319 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:29.144500 16319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:29.145933 16319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:29.146672 16354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.146853 16319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:15:29.146955 16319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "ad631a2adb5a430a9f467cfc128a703b"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "b1e6efdeec86446f721fd85b28fdc85c"
server_key_iv: "be2c0889181ccf06e4fced79873975da"
server_key_version: "encryptionkey@0"
I20260925 13:15:29.147316 16319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:29.167419 16319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:29.170014 16319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:29.170189 16319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:29.170434 16319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:29.170899 16361 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:15:29.172852 16319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:15:29.172926 16319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:15:29.172971 16319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:15:29.173622 16319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:15:29.173715 16319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 13:15:29.173766 16361 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Bootstrap starting.
I20260925 13:15:29.181062 16319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36837
I20260925 13:15:29.181460 16319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:15:29.185385 16361 log.cc:824] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:29.185765 16468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36837 every 8 connection(s)
I20260925 13:15:29.188037 16361 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:15:29.188333 16361 tablet_bootstrap.cc:492] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Bootstrap complete.
I20260925 13:15:29.189323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16319
I20260925 13:15:29.189554 16361 ts_tablet_manager.cc:1397] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260925 13:15:29.189611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37207
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46859
--webserver_port=40457
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:29.190824 16361 raft_consensus.cc:348] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.191143 16361 raft_consensus.cc:729] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad631a2adb5a430a9f467cfc128a703b, State: Initialized, Role: FOLLOWER
I20260925 13:15:29.191288 16361 consensus_queue.cc:260] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.191555 16361 ts_tablet_manager.cc:1428] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260925 13:15:29.194892 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.182229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:50351)
0925 13:15:29.183594 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.183642 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.184164 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.184630 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.184634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.184949 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:29.185139 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.185145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.186576 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.186577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.187387 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.187390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.188035 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.188036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.188182 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.188755 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:29.188763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:29.190778 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:29.192904 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:29.192909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:29.192911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:29.193231 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:29.193535 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:29.193536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:29.193536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:29.193695 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:29.194153 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:29.194160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:29.194326 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.194420 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.194625 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"thread_start_us":95,"threads_started":1}
I20260925 13:15:29.195546 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.182352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50351 (local address 127.0.98.62:46859)
0925 13:15:29.182659 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.182696 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.183766 (+ 1070us) server_negotiation.cc:407] Connection header received
0925 13:15:29.184396 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.184401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.184447 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.184597 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:29.185267 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.186446 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.187500 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.187894 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.188047 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.190917 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:29.190923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:29.190923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:29.190938 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:29.192807 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.193326 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.193328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.193329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.193442 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.193769 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.193770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.193770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.194069 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:29.195055 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.195134 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.195286 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":184,"threads_started":1}
I20260925 13:15:29.196252 16469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260925 13:15:29.196388 16469 heartbeater.cc:464] Registering TS with master...
I20260925 13:15:29.196604 16469 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260925 13:15:29.197525 16281 ts_manager.cc:194] Registered new tserver with Master: ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
I20260925 13:15:29.198872 16281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50351
I20260925 13:15:29.200079 16469 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260925 13:15:29.204896 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.199331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.0.1:50574)
0925 13:15:29.200544 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.200582 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.200649 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.200927 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.200931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.200934 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:29.201010 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.201012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.202305 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.202306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.203666 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.203669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.204607 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.204608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.204688 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.204700 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.204769 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.204818 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":100,"threads_started":1}
I20260925 13:15:29.204921 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.199440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.0.98.1:36837)
0925 13:15:29.200467 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.200472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.200724 (+ 252us) server_negotiation.cc:407] Connection header received
0925 13:15:29.200758 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.200811 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.200826 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.200895 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:29.201091 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.202217 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.203765 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.204441 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.204509 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.204563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.204797 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.204846 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":84,"threads_started":1}
I20260925 13:15:29.205607 16403 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e086b5185f463fa7e75f6876197fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50574
I20260925 13:15:29.206247 16478 tablet_replica.cc:330] stopping tablet replica
I20260925 13:15:29.206382 16478 raft_consensus.cc:2229] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:15:29.206456 16478 raft_consensus.cc:2258] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:15:29.206775 16478 ts_tablet_manager.cc:1905] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:15:29.209167 16478 ts_tablet_manager.cc:1918] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:15:29.209234 16478 log.cc:1196] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal/wals/c4e086b5185f463fa7e75f6876197fef
I20260925 13:15:29.209502 16478 ts_tablet_manager.cc:1939] T c4e086b5185f463fa7e75f6876197fef P ad631a2adb5a430a9f467cfc128a703b: Deleting consensus metadata
I20260925 13:15:29.210041 16268 catalog_manager.cc:4841] TS ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837): tablet c4e086b5185f463fa7e75f6876197fef (table test-table [id=d245b51b9f9142bc9e439e459dd74e30]) successfully deleted
W20260925 13:15:29.276419 16474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:29.276635 16474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:29.276670 16474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:29.277900 16474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:29.277966 16474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:29.278008 16474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:15:29.279551 16474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40457
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:29.279804 16474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:29.280040 16474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:29.284998 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:29.285022 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:29.285017 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:29.285164 16474 server_base.cc:1034] running on GCE node
I20260925 13:15:29.285349 16474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:29.285558 16474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:29.286701 16474 hybrid_clock.cc:648] HybridClock initialized: now 1790342129286705 us; error 45 us; skew 500 ppm
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:29.289193 16474 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:15:29.289615 16474 webserver.cc:460] Webserver started at http://127.0.98.2:40457/ using document root <none> and password file <none>
I20260925 13:15:29.289870 16474 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:29.289944 16474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:29.291302 16474 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:29.291977 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.292167 16474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:29.292268 16474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "b49845143cec411d9e6279854bcd96e7"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "68dd0d625914d12561d52a6c625727fa"
server_key_iv: "7909cb55c1310c46870a7231f989cbb1"
server_key_version: "encryptionkey@0"
I20260925 13:15:29.292616 16474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:29.301071 16474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:29.303130 16474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:29.303292 16474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:29.303524 16474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:29.303957 16500 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 13:15:29.305401 16500 ts_tablet_manager.cc:1852] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 13:15:29.305725 16500 ts_tablet_manager.cc:1905] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:15:29.308135 16500 ts_tablet_manager.cc:1918] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:15:29.308260 16500 ts_tablet_manager.cc:1939] T c4e086b5185f463fa7e75f6876197fef P b49845143cec411d9e6279854bcd96e7: Deleting consensus metadata
I20260925 13:15:29.308485 16474 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 13:15:29.308547 16474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 13:15:29.308605 16474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:15:29.308634 16474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.314503 16474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37207
I20260925 13:15:29.314554 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37207 every 8 connection(s)
I20260925 13:15:29.314838 16474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:15:29.316212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16474
I20260925 13:15:29.316509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45607
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46859
--webserver_port=40671
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:29.324659 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.315513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:35533)
0925 13:15:29.315685 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.315720 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.316177 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.316475 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.316479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.316813 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:29.317006 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.317011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.317824 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.317825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.318754 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.318757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.319133 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.319134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.319282 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.319808 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:29.319819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:29.321400 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:29.323300 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:29.323305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:29.323307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:29.323539 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:29.323765 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:29.323765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:29.323766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:29.323858 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:29.324151 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:29.324155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:29.324289 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.324381 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.324496 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260925 13:15:29.324664 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.315512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35533 (local address 127.0.98.62:46859)
0925 13:15:29.315568 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.315571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.315813 (+ 242us) server_negotiation.cc:407] Connection header received
0925 13:15:29.316292 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.316293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.316310 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.316398 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:29.317103 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.317727 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.318844 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.319049 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.319106 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.321495 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:29.321497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:29.321498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:29.321507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:29.323214 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.323623 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.323625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.323626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.323690 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.323924 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.323925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.323925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.324081 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:29.324152 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.324418 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.324522 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:15:29.325239 16608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260925 13:15:29.325345 16608 heartbeater.cc:464] Registering TS with master...
I20260925 13:15:29.325558 16608 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260925 13:15:29.326015 16281 ts_manager.cc:194] Registered new tserver with Master: b49845143cec411d9e6279854bcd96e7 (127.0.98.2:37207)
I20260925 13:15:29.326566 16281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35533
W20260925 13:15:29.399612 16611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:29.399833 16611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:29.399859 16611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:29.401113 16611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:29.401155 16611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:29.401181 16611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:15:29.402603 16611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40671
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:29.402788 16611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:29.402962 16611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:29.407970 16617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:29.408093 16611 server_base.cc:1034] running on GCE node
W20260925 13:15:29.407967 16619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:29.407966 16616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:29.408396 16611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:29.408632 16611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:29.409813 16611 hybrid_clock.cc:648] HybridClock initialized: now 1790342129409799 us; error 42 us; skew 500 ppm
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:29.412279 16611 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:15:29.412775 16611 webserver.cc:460] Webserver started at http://127.0.98.3:40671/ using document root <none> and password file <none>
I20260925 13:15:29.413029 16611 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:29.413111 16611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:29.414532 16611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:15:29.415158 16626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.415345 16611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:15:29.415433 16611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "bf98ce4c32594601a3ef73f751033caf"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "0a6b34668fa774ccd263ea84708cf963"
server_key_iv: "c7d8bef07d6039bd3e66432a1d0a947b"
server_key_version: "encryptionkey@0"
I20260925 13:15:29.415812 16611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:29.430065 16611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:29.432205 16611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:29.432410 16611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:29.432667 16611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:29.432967 16611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:15:29.433041 16611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.433104 16611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:15:29.433135 16611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.439167 16611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45607
I20260925 13:15:29.439215 16739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45607 every 8 connection(s)
I20260925 13:15:29.439587 16611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:15:29.441650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16611
Sep 25 13:15:29 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:15:29.449203 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.442147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.0.98.62:46859)
0925 13:15:29.442309 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.442311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.442363 (+ 52us) server_negotiation.cc:407] Connection header received
0925 13:15:29.442443 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.442444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.442455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.442558 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:29.443129 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.443625 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.444593 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.444833 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.444885 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.446320 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:29.446321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:29.446322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:29.446328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:29.448016 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.448422 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.448424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.448424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.448475 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.448681 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.448681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.448682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.448780 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:29.448966 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.449006 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.449087 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
I20260925 13:15:29.450161 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.440350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52725 (local address 127.0.98.62:46859)
0925 13:15:29.440432 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.440433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.440713 (+ 280us) server_negotiation.cc:407] Connection header received
0925 13:15:29.441209 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.441210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.441221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.441283 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:29.442561 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.443111 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.444358 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.444569 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.444656 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.447212 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:29.447213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:29.447214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:29.447220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:29.448756 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.449125 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.449126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.449127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.449192 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.449516 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.449517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.449517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.449622 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:29.449721 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.450001 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.450075 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 13:15:29.450371 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.440356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:52725)
0925 13:15:29.440580 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.440619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.441099 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.441377 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.441381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.442152 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:29.442469 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.442475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.443210 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.443210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.444264 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.444266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.444669 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.444670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.444769 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.445303 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:29.445353 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:29.447120 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:29.448827 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:29.448833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:29.448836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:29.449050 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:29.449277 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:29.449278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:29.449279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:29.449421 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:29.449716 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:29.449722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:29.449870 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.449982 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.450139 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260925 13:15:29.450805 16740 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260925 13:15:29.450884 16740 heartbeater.cc:464] Registering TS with master...
I20260925 13:15:29.451016 16740 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260925 13:15:29.451368 16281 ts_manager.cc:194] Registered new tserver with Master: bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607)
I20260925 13:15:29.451741 16281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52725
I20260925 13:15:29.460857 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:15:29.463732 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.461239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.0.98.62:46859)
0925 13:15:29.461328 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.461330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.461416 (+ 86us) server_negotiation.cc:407] Connection header received
0925 13:15:29.461469 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.461470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.461512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.461604 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:15:29.461834 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.462305 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.463095 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.463293 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.463360 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.463498 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:15:29.463545 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 13:15:29.463680 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42248: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 13:15:29.463825 16472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42248: token expired
W20260925 13:15:29.463948 16209 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46859)
I20260925 13:15:29.469743 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.464204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.98.62:46859)
0925 13:15:29.464246 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.464248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.464289 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:15:29.464337 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.464338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.464345 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.464392 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:29.464860 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.465315 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.466074 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.466245 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.466314 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.467522 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:29.467523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:29.467523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:29.467530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:29.468758 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.469054 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.469055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.469055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.469105 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:29.469260 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:29.469261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:29.469261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:29.469384 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:29.469456 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.469541 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.469596 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:15:29.470894 16281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:15:29.472553 16281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:15:29.474753 16403 tablet_service.cc:1467] Processing CreateTablet for tablet 71551d9484ac4206b034e0cf33f481d6 (DEFAULT_TABLE table=test-table [id=f2b98367d7d2447399f718f6c7353c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:15:29.474974 16403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71551d9484ac4206b034e0cf33f481d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:29.476588 16361 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Bootstrap starting.
I20260925 13:15:29.476909 16361 tablet_bootstrap.cc:654] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:29.477802 16361 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: No bootstrap required, opened a new log
I20260925 13:15:29.477867 16361 ts_tablet_manager.cc:1397] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:15:29.477980 16361 raft_consensus.cc:348] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.478055 16361 raft_consensus.cc:374] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:29.478109 16361 raft_consensus.cc:729] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad631a2adb5a430a9f467cfc128a703b, State: Initialized, Role: FOLLOWER
I20260925 13:15:29.478164 16361 consensus_queue.cc:260] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.478271 16361 ts_tablet_manager.cc:1428] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:29.479135 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.475039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.3:45607)
0925 13:15:29.475349 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.475354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.475363 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:15:29.475401 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.475403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.475417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.475493 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:29.475683 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.476730 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.478036 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.478725 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.478824 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.478885 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.478987 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.479031 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
I20260925 13:15:29.479202 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.474492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37207 (local address 127.0.0.1:54970)
0925 13:15:29.474527 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.474536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.474570 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.475142 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.475143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.475145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:29.475187 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.475188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.476214 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.476215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.477226 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.477229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.478966 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.478967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.479028 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.479032 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.479089 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.479133 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 13:15:29.479216 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.474746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.0.98.2:37207)
0925 13:15:29.474955 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.474960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.474971 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:15:29.475014 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.475016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.475034 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.475144 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:29.475251 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.476131 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.477337 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.478881 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.478967 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.479023 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.479104 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.479138 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260925 13:15:29.479328 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.474585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45607 (local address 127.0.0.1:47462)
0925 13:15:29.474757 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.474768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.474822 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.475582 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.475583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.475585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:29.475620 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.475621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.476829 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.476829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.477932 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.477935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.478826 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.478827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.478891 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.478895 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.479215 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.479267 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260925 13:15:29.479908 16542 tablet_service.cc:1467] Processing CreateTablet for tablet 71551d9484ac4206b034e0cf33f481d6 (DEFAULT_TABLE table=test-table [id=f2b98367d7d2447399f718f6c7353c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:15:29.480116 16674 tablet_service.cc:1467] Processing CreateTablet for tablet 71551d9484ac4206b034e0cf33f481d6 (DEFAULT_TABLE table=test-table [id=f2b98367d7d2447399f718f6c7353c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:15:29.480222 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71551d9484ac4206b034e0cf33f481d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:29.480415 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71551d9484ac4206b034e0cf33f481d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:29.482421 16500 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Bootstrap starting.
I20260925 13:15:29.482939 16500 tablet_bootstrap.cc:654] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:29.483397 16500 log.cc:824] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:29.483639 16750 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Bootstrap starting.
I20260925 13:15:29.484139 16500 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: No bootstrap required, opened a new log
I20260925 13:15:29.484211 16500 ts_tablet_manager.cc:1397] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:15:29.484282 16750 tablet_bootstrap.cc:654] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Neither blocks nor log segments found. Creating new log.
I20260925 13:15:29.484767 16750 log.cc:824] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:29.485430 16750 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: No bootstrap required, opened a new log
I20260925 13:15:29.485482 16750 ts_tablet_manager.cc:1397] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:15:29.485756 16500 raft_consensus.cc:348] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.485914 16500 raft_consensus.cc:374] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:29.485958 16500 raft_consensus.cc:729] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49845143cec411d9e6279854bcd96e7, State: Initialized, Role: FOLLOWER
I20260925 13:15:29.486086 16500 consensus_queue.cc:260] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.486352 16500 ts_tablet_manager.cc:1428] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:15:29.486382 16608 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260925 13:15:29.487689 16750 raft_consensus.cc:348] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.487815 16750 raft_consensus.cc:374] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:15:29.487835 16750 raft_consensus.cc:729] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf98ce4c32594601a3ef73f751033caf, State: Initialized, Role: FOLLOWER
I20260925 13:15:29.487965 16750 consensus_queue.cc:260] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.488230 16750 ts_tablet_manager.cc:1428] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:15:29.488250 16740 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260925 13:15:29.563640 16754 raft_consensus.cc:482] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:15:29.563766 16754 raft_consensus.cc:504] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.564092 16754 leader_election.cc:288] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b49845143cec411d9e6279854bcd96e7 (127.0.98.2:37207), ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
W20260925 13:15:29.565346 16609 tablet.cc:2369] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:15:29.567082 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.564451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53795 (local address 127.0.98.2:37207)
0925 13:15:29.564498 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.564500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.564508 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:15:29.564540 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.564541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.564552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.564617 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:29.564767 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.565337 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.566349 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.566734 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.566818 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.566828 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.566980 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.567015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:15:29.567083 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.564296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37207 (local address 127.0.98.3:53795)
0925 13:15:29.564354 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.564369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.564445 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.564620 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.564621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.564622 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:29.564704 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.564706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.565420 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.565421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.566257 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.566259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.566816 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.566817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.566908 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.566910 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.566975 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.567013 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 13:15:29.567237 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.564405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39693 (local address 127.0.98.1:36837)
0925 13:15:29.564451 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.564453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.564683 (+ 230us) server_negotiation.cc:407] Connection header received
0925 13:15:29.564729 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.564730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.564743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.564796 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:15:29.564968 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.565544 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.566509 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.566879 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.566969 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.566978 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.567136 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.567176 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:15:29.567235 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.564304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.98.3:39693)
0925 13:15:29.564612 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.564631 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:29.564685 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:29.564818 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:29.564819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:29.564821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:15:29.564896 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.564897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.565619 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.565619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.566412 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:29.566413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.566973 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.566974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:29.567053 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.567056 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:15:29.567117 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 13:15:29.567156 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":130,"thread_start_us":126,"threads_started":1}
I20260925 13:15:29.567553 16423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71551d9484ac4206b034e0cf33f481d6" candidate_uuid: "bf98ce4c32594601a3ef73f751033caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad631a2adb5a430a9f467cfc128a703b" is_pre_election: true
I20260925 13:15:29.567638 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71551d9484ac4206b034e0cf33f481d6" candidate_uuid: "bf98ce4c32594601a3ef73f751033caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49845143cec411d9e6279854bcd96e7" is_pre_election: true
I20260925 13:15:29.567739 16423 raft_consensus.cc:2454] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf98ce4c32594601a3ef73f751033caf in term 0.
I20260925 13:15:29.567752 16562 raft_consensus.cc:2454] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf98ce4c32594601a3ef73f751033caf in term 0.
I20260925 13:15:29.568001 16627 leader_election.cc:302] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b49845143cec411d9e6279854bcd96e7, bf98ce4c32594601a3ef73f751033caf; no voters:
I20260925 13:15:29.568132 16754 raft_consensus.cc:2781] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:15:29.568221 16754 raft_consensus.cc:482] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:15:29.568259 16754 raft_consensus.cc:3037] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:29.568955 16754 raft_consensus.cc:504] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.569106 16754 leader_election.cc:288] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [CANDIDATE]: Term 1 election: Requested vote from peers b49845143cec411d9e6279854bcd96e7 (127.0.98.2:37207), ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
I20260925 13:15:29.569270 16423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71551d9484ac4206b034e0cf33f481d6" candidate_uuid: "bf98ce4c32594601a3ef73f751033caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad631a2adb5a430a9f467cfc128a703b"
I20260925 13:15:29.569271 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71551d9484ac4206b034e0cf33f481d6" candidate_uuid: "bf98ce4c32594601a3ef73f751033caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49845143cec411d9e6279854bcd96e7"
I20260925 13:15:29.569367 16562 raft_consensus.cc:3037] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:29.569375 16423 raft_consensus.cc:3037] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:15:29.570214 16562 raft_consensus.cc:2454] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf98ce4c32594601a3ef73f751033caf in term 1.
I20260925 13:15:29.570405 16627 leader_election.cc:302] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b49845143cec411d9e6279854bcd96e7, bf98ce4c32594601a3ef73f751033caf; no voters:
I20260925 13:15:29.570497 16754 raft_consensus.cc:2781] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:15:29.570628 16754 raft_consensus.cc:686] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 1 LEADER]: Becoming Leader. State: Replica: bf98ce4c32594601a3ef73f751033caf, State: Running, Role: LEADER
I20260925 13:15:29.570735 16423 raft_consensus.cc:2454] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf98ce4c32594601a3ef73f751033caf in term 1.
I20260925 13:15:29.570729 16754 consensus_queue.cc:237] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:29.571633 16281 catalog_manager.cc:5495] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf reported cstate change: term changed from 0 to 1, leader changed from <none> to bf98ce4c32594601a3ef73f751033caf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bf98ce4c32594601a3ef73f751033caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260925 13:15:29.598320 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.595847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.98.3:45607)
0925 13:15:29.595923 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:29.595925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:29.595933 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:15:29.595968 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:29.595969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:29.595989 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:29.596048 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:15:29.596174 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.596664 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.597443 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:29.597647 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:29.597733 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:29.597926 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:15:29.598030 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:15:29.598092 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:29.598131 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 13:15:29.598150 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:15:29.600257 16562 raft_consensus.cc:1261] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Refusing update from remote peer bf98ce4c32594601a3ef73f751033caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:15:29.600431 16423 raft_consensus.cc:1261] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Refusing update from remote peer bf98ce4c32594601a3ef73f751033caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:15:29.600641 16754 consensus_queue.cc:1035] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:15:29.600768 16756 consensus_queue.cc:1035] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:15:29.603636 16761 mvcc.cc:205] Tried to move back new op lower bound from 7333241362839896064 to 7333241362722856960. Current Snapshot: MvccSnapshot[applied={T|T < 7333241362839896064 or (T in {7333241362839896064})}]
I20260925 13:15:29.605929 16281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:15:29.606012 16281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:15:29.606854 16281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Sending DeleteTablet for 3 replicas of tablet 71551d9484ac4206b034e0cf33f481d6
I20260925 13:15:29.607168 16403 tablet_service.cc:1514] Processing DeleteTablet for tablet 71551d9484ac4206b034e0cf33f481d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50574
I20260925 13:15:29.607187 16542 tablet_service.cc:1514] Processing DeleteTablet for tablet 71551d9484ac4206b034e0cf33f481d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54970
I20260925 13:15:29.607295 16674 tablet_service.cc:1514] Processing DeleteTablet for tablet 71551d9484ac4206b034e0cf33f481d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:15:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47462
I20260925 13:15:29.607518 16478 tablet_replica.cc:330] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: stopping tablet replica
I20260925 13:15:29.607586 16478 raft_consensus.cc:2229] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:15:29.607604 16770 tablet_replica.cc:330] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: stopping tablet replica
I20260925 13:15:29.607652 16478 raft_consensus.cc:2258] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:15:29.607764 16770 raft_consensus.cc:2229] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:15:29.607934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 16319
I20260925 13:15:29.607929 16770 raft_consensus.cc:2258] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:15:29.608021 16478 ts_tablet_manager.cc:1905] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:15:29.609031 16770 ts_tablet_manager.cc:1905] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:15:29.611188 16770 ts_tablet_manager.cc:1918] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:15:29.611282 16770 log.cc:1196] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/71551d9484ac4206b034e0cf33f481d6
I20260925 13:15:29.611542 16770 ts_tablet_manager.cc:1939] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Deleting consensus metadata
I20260925 13:15:29.611974 16265 catalog_manager.cc:4841] TS bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607): tablet 71551d9484ac4206b034e0cf33f481d6 (table test-table [id=f2b98367d7d2447399f718f6c7353c0d]) successfully deleted
I20260925 13:15:29.613826 16769 tablet_replica.cc:330] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: stopping tablet replica
I20260925 13:15:29.613972 16769 raft_consensus.cc:2229] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:15:29.614090 16769 raft_consensus.cc:2258] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:15:29.614521 16769 ts_tablet_manager.cc:1905] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 13:15:29.614728 16268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36837 (error 108)
I20260925 13:15:29.615135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 16474
I20260925 13:15:29.615705 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.615055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.0.1:50586)
0925 13:15:29.615621 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.615670 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537}
W20260925 13:15:29.619741 16268 catalog_manager.cc:4584] TS ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71551d9484ac4206b034e0cf33f481d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36837: connect: Connection refused (error 111)
I20260925 13:15:29.621500 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:29.621389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37207 (local address 127.0.0.1:54984)
0925 13:15:29.621435 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:29.621459 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 13:15:29.623158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 16611
I20260925 13:15:29.627702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 16248
I20260925 13:15:49.633255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--webserver_interface=127.0.98.62
--webserver_port=33629
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:49.715541 16772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:49.715755 16772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:49.715778 16772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:49.716954 16772 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 13:15:49.716995 16772 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 13:15:49.717008 16772 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 13:15:49.717020 16772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:15:49.717033 16772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:49.717094 16772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:49.717111 16772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:15:49.717135 16772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:15:49.718696 16772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:49.718957 16772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:49.719208 16772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:49.724502 16780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:49.724462 16777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:49.724483 16778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:49.724614 16772 server_base.cc:1034] running on GCE node
I20260925 13:15:49.724855 16772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:49.725157 16772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:49.726326 16772 hybrid_clock.cc:648] HybridClock initialized: now 1790342149726314 us; error 26 us; skew 500 ppm
Sep 25 13:15:49 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:49.735496 16772 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:15:49.735934 16772 webserver.cc:460] Webserver started at http://127.0.98.62:33629/ using document root <none> and password file <none>
I20260925 13:15:49.736189 16772 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:49.736266 16772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:49.737623 16772 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:49.738337 16787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:49.738559 16772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:49.738651 16772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "9ac284ee121f4d9592df3dbaa491c0c0"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "9e1b8c762cedc09bac8764c630789dc9"
server_key_iv: "ce23ab187eb9483b97584c52e6b3a3e6"
server_key_version: "encryptionkey@0"
I20260925 13:15:49.738991 16772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:49.748338 16772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:49.750417 16772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:49.750521 16772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:49.754541 16772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46859
I20260925 13:15:49.754575 16839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46859 every 8 connection(s)
I20260925 13:15:49.754958 16772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:15:49.756376 16840 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:15:49.757071 16840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap starting.
I20260925 13:15:49.758141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16772
I20260925 13:15:49.758466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36837
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46859
--webserver_port=46075
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:15:49.766232 16840 log.cc:824] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:49.768395 16840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:15:49.768715 16840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Bootstrap complete.
I20260925 13:15:49.770970 16840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:49.771178 16840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac284ee121f4d9592df3dbaa491c0c0, State: Initialized, Role: FOLLOWER
I20260925 13:15:49.771345 16840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:49.771407 16840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:15:49.771476 16840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:15:49.771517 16840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 13:15:49.772539 16840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:49.772670 16840 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac284ee121f4d9592df3dbaa491c0c0; no voters:
I20260925 13:15:49.772883 16840 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 13:15:49.772917 16844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 13:15:49.773031 16844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [term 3 LEADER]: Becoming Leader. State: Replica: 9ac284ee121f4d9592df3dbaa491c0c0, State: Running, Role: LEADER
I20260925 13:15:49.773116 16844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260925 13:15:49.773144 16840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:15:49.773801 16846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac284ee121f4d9592df3dbaa491c0c0. Latest consensus state: current_term: 3 leader_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260925 13:15:49.773890 16846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:15:49.774021 16845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac284ee121f4d9592df3dbaa491c0c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260925 13:15:49.774071 16845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:15:49.774185 16853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:15:49.774917 16853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:15:49.775025 16853 catalog_manager.cc:1259] Loaded cluster ID: 1f7c00e498e1477e94d5d52959d42ea6
I20260925 13:15:49.775255 16853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:15:49.776551 16853 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:15:49.776724 16853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Loaded TSK: 0
I20260925 13:15:49.776768 16853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ac284ee121f4d9592df3dbaa491c0c0: Loaded TSK: 1
F20260925 13:15:49.776999 16853 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff7c20449ad google::LogMessage::Fail() at ??:0
@ 0x7ff7c204788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff7c20444de google::LogMessage::Flush() at ??:0
@ 0x7ff7c2045a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff7c47b1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff7c47aed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff7c9f9e1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff7c9f9e48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff7c9f9e594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff7c9f64c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff7c9f6ead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff7c3f120d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff7c3f07954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff7c4d6b6db start_thread at ??:0
@ 0x7ff7c0b4261f clone at ??:0
W20260925 13:15:49.862810 16842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:49.863015 16842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:49.863039 16842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:49.864177 16842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:49.864216 16842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:49.864243 16842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:15:49.865622 16842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46075
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:49.865945 16842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:49.866173 16842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:49.872069 16881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:49.872044 16884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:49.872062 16882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:49.872294 16842 server_base.cc:1034] running on GCE node
I20260925 13:15:49.872545 16842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:49.872819 16842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:49.873993 16842 hybrid_clock.cc:648] HybridClock initialized: now 1790342149873966 us; error 45 us; skew 500 ppm
Sep 25 13:15:49 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:49.876546 16842 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:15:49.876986 16842 webserver.cc:460] Webserver started at http://127.0.98.1:46075/ using document root <none> and password file <none>
I20260925 13:15:49.877223 16842 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:49.877296 16842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:49.878793 16842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:49.879608 16891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:49.879842 16842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:49.879946 16842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "ad631a2adb5a430a9f467cfc128a703b"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "b1e6efdeec86446f721fd85b28fdc85c"
server_key_iv: "be2c0889181ccf06e4fced79873975da"
server_key_version: "encryptionkey@0"
I20260925 13:15:49.880306 16842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:49.904549 16842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:49.906929 16842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:49.907102 16842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:49.907356 16842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:49.907845 16898 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:15:49.909868 16842 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:15:49.909937 16842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:15:49.909986 16842 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:15:49.910621 16842 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:15:49.910683 16842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:15:49.910770 16898 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Bootstrap starting.
I20260925 13:15:49.912709 16898 log.cc:824] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:49.916091 16898 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:15:49.916435 16898 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Bootstrap complete.
I20260925 13:15:49.916607 16898 ts_tablet_manager.cc:1397] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 13:15:49.917693 16842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36837
I20260925 13:15:49.918068 16842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:15:49.918788 17006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36837 every 8 connection(s)
I20260925 13:15:49.918814 16898 raft_consensus.cc:348] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:49.919162 16898 raft_consensus.cc:729] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad631a2adb5a430a9f467cfc128a703b, State: Initialized, Role: FOLLOWER
I20260925 13:15:49.919301 16898 consensus_queue.cc:260] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:49.919549 16898 ts_tablet_manager.cc:1428] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260925 13:15:49.924181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 16842
I20260925 13:15:49.924502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37207
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46859
--webserver_port=40457
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:49.999989 17009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:15:49.918880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:35447)
0925 13:15:49.919069 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:49.919111 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:49.919264 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:49.999745 (+ 80481us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
W20260925 13:15:50.000317 17007 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46859 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46859: Client connection negotiation failed: client connection to 127.0.98.62:46859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 13:15:50.000677 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.000553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:42459)
0925 13:15:50.000611 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.000658 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:15:50.001137 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.001011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:59377)
0925 13:15:50.001084 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.001115 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 13:15:50.001356 17007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:15:50.037078 17011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:50.037320 17011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:50.037349 17011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:50.038671 17011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:50.038724 17011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:50.038755 17011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:15:50.040349 17011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40457
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:50.040611 17011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:50.040889 17011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:50.046768 17016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:50.046857 17011 server_base.cc:1034] running on GCE node
W20260925 13:15:50.046789 17019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:50.046962 17017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:50.047225 17011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:50.047583 17011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:50.048785 17011 hybrid_clock.cc:648] HybridClock initialized: now 1790342150048746 us; error 56 us; skew 500 ppm
Sep 25 13:15:50 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:50.051627 17011 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:15:50.052120 17011 webserver.cc:460] Webserver started at http://127.0.98.2:40457/ using document root <none> and password file <none>
I20260925 13:15:50.052384 17011 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:50.052459 17011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:50.053865 17011 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:50.054631 17026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:50.054800 17011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:15:50.054903 17011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "b49845143cec411d9e6279854bcd96e7"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "68dd0d625914d12561d52a6c625727fa"
server_key_iv: "7909cb55c1310c46870a7231f989cbb1"
server_key_version: "encryptionkey@0"
I20260925 13:15:50.055274 17011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:50.064213 17011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:50.066737 17011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:50.066906 17011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:50.067166 17011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:50.067638 17033 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:15:50.069067 17011 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:15:50.069121 17011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:15:50.069147 17011 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:15:50.069857 17011 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:15:50.069902 17011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:50.069967 17033 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Bootstrap starting.
I20260925 13:15:50.072052 17033 log.cc:824] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:50.075543 17033 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:15:50.075910 17033 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Bootstrap complete.
I20260925 13:15:50.076081 17033 ts_tablet_manager.cc:1397] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 13:15:50.077275 17011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37207
I20260925 13:15:50.077456 17141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37207 every 8 connection(s)
I20260925 13:15:50.077667 17011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:15:50.078306 17033 raft_consensus.cc:348] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:50.078630 17033 raft_consensus.cc:729] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49845143cec411d9e6279854bcd96e7, State: Initialized, Role: FOLLOWER
I20260925 13:15:50.078768 17033 consensus_queue.cc:260] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:50.078809 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.078372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:37631)
0925 13:15:50.078569 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.078650 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260925 13:15:50.079053 17033 ts_tablet_manager.cc:1428] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260925 13:15:50.079077 17142 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46859 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46859: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
I20260925 13:15:50.079392 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.079276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:43949)
0925 13:15:50.079347 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.079376 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 13:15:50.079662 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.079570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:43229)
0925 13:15:50.079623 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.079644 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 13:15:50.079767 17142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:15:50.080612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17011
I20260925 13:15:50.080852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45607
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46859
--webserver_port=40671
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:15:50.160578 17146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:15:50.160758 17146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:15:50.160781 17146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:15:50.161839 17146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:15:50.161871 17146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:15:50.161896 17146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:15:50.163218 17146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40671
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:15:50.163394 17146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:15:50.163556 17146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:15:50.168453 17152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:15:50.168439 17154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:50.168509 17146 server_base.cc:1034] running on GCE node
W20260925 13:15:50.168455 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:15:50.168892 17146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:15:50.169137 17146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:15:50.170310 17146 hybrid_clock.cc:648] HybridClock initialized: now 1790342150170294 us; error 33 us; skew 500 ppm
Sep 25 13:15:50 dist-test-slave-2d0s krb5kdc[15702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:15:50.172629 17146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:15:50.173054 17146 webserver.cc:460] Webserver started at http://127.0.98.3:40671/ using document root <none> and password file <none>
I20260925 13:15:50.173290 17146 fs_manager.cc:359] Metadata directory not provided
I20260925 13:15:50.173362 17146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:15:50.174693 17146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:15:50.175282 17161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:50.175451 17146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:15:50.175549 17146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "bf98ce4c32594601a3ef73f751033caf"
format_stamp: "Formatted at 2026-09-25 13:15:08 on dist-test-slave-2d0s"
server_key: "0a6b34668fa774ccd263ea84708cf963"
server_key_iv: "c7d8bef07d6039bd3e66432a1d0a947b"
server_key_version: "encryptionkey@0"
I20260925 13:15:50.175871 17146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:15:50.224793 17146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:15:50.226840 17146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:15:50.227007 17146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:15:50.227248 17146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:15:50.227594 17146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:15:50.227653 17146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:50.227694 17146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:15:50.227715 17146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:15:50.233422 17146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45607
I20260925 13:15:50.233472 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45607 every 8 connection(s)
I20260925 13:15:50.233835 17146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:15:50.234853 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.234471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:51057)
0925 13:15:50.234629 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.234713 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
W20260925 13:15:50.235283 17275 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46859 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46859: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
I20260925 13:15:50.235584 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.235495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:47217)
0925 13:15:50.235541 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.235566 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:15:50.235881 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:50.235793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:48171)
0925 13:15:50.235847 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:50.235868 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:15:50.236155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17146
W20260925 13:15:50.236378 17275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:15:51.002053 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.001822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:57589)
0925 13:15:51.001958 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:51.001994 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260925 13:15:51.080294 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.080074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:51983)
0925 13:15:51.080215 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:51.080247 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260925 13:15:51.236943 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.236743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:50179)
0925 13:15:51.236860 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:51.236894 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260925 13:15:51.556594 17282 raft_consensus.cc:482] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:15:51.556773 17282 raft_consensus.cc:504] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:51.557117 17282 leader_election.cc:288] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49845143cec411d9e6279854bcd96e7 (127.0.98.2:37207), bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607)
Sep 25 13:15:51 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 13:15:51 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 13:15:51.568063 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.557375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37207 (local address 127.0.98.1:33599)
0925 13:15:51.557514 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:51.557526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:51.557582 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:51.558242 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:51.558245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:51.558575 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:51.559058 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:51.559061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.560194 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.560195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:51.561186 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:51.561188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.561850 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.561851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:51.561976 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:51.562679 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:51.562684 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:51.564134 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:51.566505 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:51.566511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:51.566514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:51.566745 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:51.567031 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:51.567032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:51.567033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:51.567133 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:51.567523 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:51.567529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:51.567676 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 13:15:51.567790 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 13:15:51.567965 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":60,"thread_start_us":82,"threads_started":1}
I20260925 13:15:51.568075 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.557501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33599 (local address 127.0.98.2:37207)
0925 13:15:51.557760 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:51.557770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:51.557791 (+ 21us) server_negotiation.cc:407] Connection header received
0925 13:15:51.557957 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:51.557959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:51.558012 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:51.558186 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:51.559171 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.560067 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.561296 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.561746 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.561876 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:51.564270 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:51.564274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:51.564274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:51.564285 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:51.566402 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:51.566848 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:51.566851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:51.566852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:51.566951 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:51.567205 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:51.567206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:51.567207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:51.567438 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:51.567530 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:51.567840 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 13:15:51.567961 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
I20260925 13:15:51.568162 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.557759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52979 (local address 127.0.98.3:45607)
0925 13:15:51.557895 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:51.557902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:51.557931 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:15:51.558483 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:51.558485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:51.558521 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:51.558688 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:51.559182 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.560070 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.561302 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.561774 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.561904 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:51.564591 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:51.564595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:51.564595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:51.564606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:51.566703 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:51.567076 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:51.567078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:51.567079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:51.567167 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:51.567384 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:51.567384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:51.567385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:51.567607 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:51.567705 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:51.567900 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 13:15:51.568046 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 13:15:51.568195 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.557596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45607 (local address 127.0.98.1:52979)
0925 13:15:51.557707 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:51.557717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:51.557759 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:51.558682 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:51.558684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:51.558923 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:51.559053 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:51.559059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.560231 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.560232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:51.561184 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:51.561187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:51.561860 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:51.561861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:51.561978 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:51.562591 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:51.562600 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:51.564478 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:51.566805 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:51.566806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:51.566807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:51.566976 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:51.567244 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:51.567245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:51.567245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:51.567311 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:51.567701 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:51.567702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:51.567746 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:15:51.567841 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 13:15:51.568048 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":180,"thread_start_us":48,"threads_started":1}
I20260925 13:15:51.569204 17096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71551d9484ac4206b034e0cf33f481d6" candidate_uuid: "ad631a2adb5a430a9f467cfc128a703b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b49845143cec411d9e6279854bcd96e7" is_pre_election: true
I20260925 13:15:51.569362 17096 raft_consensus.cc:2454] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad631a2adb5a430a9f467cfc128a703b in term 1.
W20260925 13:15:51.569527 16892 leader_election.cc:341] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607): Not found: Tablet not found: 71551d9484ac4206b034e0cf33f481d6
I20260925 13:15:51.569643 16892 leader_election.cc:302] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad631a2adb5a430a9f467cfc128a703b, b49845143cec411d9e6279854bcd96e7; no voters: bf98ce4c32594601a3ef73f751033caf
I20260925 13:15:51.569780 17282 raft_consensus.cc:2781] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 13:15:51.569831 17282 raft_consensus.cc:482] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:15:51.569859 17282 raft_consensus.cc:3037] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:15:51.571241 17282 raft_consensus.cc:504] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:51.571393 17282 leader_election.cc:288] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [CANDIDATE]: Term 2 election: Requested vote from peers b49845143cec411d9e6279854bcd96e7 (127.0.98.2:37207), bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607)
I20260925 13:15:51.571559 17096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71551d9484ac4206b034e0cf33f481d6" candidate_uuid: "ad631a2adb5a430a9f467cfc128a703b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b49845143cec411d9e6279854bcd96e7"
I20260925 13:15:51.571664 17096 raft_consensus.cc:3037] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 1 FOLLOWER]: Advancing to term 2
W20260925 13:15:51.571676 16892 leader_election.cc:341] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607): Not found: Tablet not found: 71551d9484ac4206b034e0cf33f481d6
I20260925 13:15:51.572813 17096 raft_consensus.cc:2454] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad631a2adb5a430a9f467cfc128a703b in term 2.
I20260925 13:15:51.573045 16892 leader_election.cc:302] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad631a2adb5a430a9f467cfc128a703b, b49845143cec411d9e6279854bcd96e7; no voters: bf98ce4c32594601a3ef73f751033caf
I20260925 13:15:51.573139 17282 raft_consensus.cc:2781] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:15:51.573248 17282 raft_consensus.cc:686] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [term 2 LEADER]: Becoming Leader. State: Replica: ad631a2adb5a430a9f467cfc128a703b, State: Running, Role: LEADER
I20260925 13:15:51.573339 17282 consensus_queue.cc:237] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:51.573643 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:51.573512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:56191)
0925 13:15:51.573560 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:51.573620 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 13:15:52.022962 16892 consensus_peers.cc:487] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b -> Peer bf98ce4c32594601a3ef73f751033caf (127.0.98.3:45607): Couldn't send request to peer bf98ce4c32594601a3ef73f751033caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71551d9484ac4206b034e0cf33f481d6. This is attempt 1: this message will repeat every 5th retry.
I20260925 13:15:52.080929 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.080737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:43823)
0925 13:15:52.080853 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.080895 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260925 13:15:52.095846 17096 raft_consensus.cc:1261] T 71551d9484ac4206b034e0cf33f481d6 P b49845143cec411d9e6279854bcd96e7 [term 2 FOLLOWER]: Refusing update from remote peer ad631a2adb5a430a9f467cfc128a703b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 13:15:52.096279 17282 consensus_queue.cc:1035] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 13:15:52.097374 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.097250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:60993)
0925 13:15:52.097308 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.097342 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:15:52.098367 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.098162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:37967)
0925 13:15:52.098287 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.098322 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":47,"threads_started":1}
I20260925 13:15:52.098455 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.098344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:48571)
0925 13:15:52.098397 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.098425 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:15:52.237634 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.237382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:52729)
0925 13:15:52.237550 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.237591 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260925 13:15:52.464321 17297 ts_tablet_manager.cc:927] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Initiating tablet copy from peer ad631a2adb5a430a9f467cfc128a703b (127.0.98.1:36837)
I20260925 13:15:52.464632 17297 tablet_copy_client.cc:316] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:36837
Sep 25 13:15:52 dist-test-slave-2d0s krb5kdc[15702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:15:52.472666 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.464901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36837 (local address 127.0.98.3:48211)
0925 13:15:52.464967 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.464987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 13:15:52.465069 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:15:52.465348 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:15:52.465353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:15:52.465641 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:15:52.465794 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:52.465797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:52.466644 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:52.466645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:52.467236 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:15:52.467238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:15:52.467685 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:15:52.467685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:15:52.467739 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:52.468250 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:15:52.468254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:15:52.469655 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:15:52.471648 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:52.471655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:52.471657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:52.471847 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:52.472045 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:15:52.472045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:15:52.472049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:15:52.472107 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:15:52.472370 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:15:52.472376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:15:52.472420 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:15:52.472488 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:15:52.472575 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 13:15:52.472682 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.465034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48211 (local address 127.0.98.1:36837)
0925 13:15:52.465198 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 13:15:52.465203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:15:52.465214 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:15:52.465250 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:15:52.465251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:15:52.465264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:15:52.465328 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:15:52.465867 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:52.466556 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:52.467312 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:15:52.467612 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:15:52.467665 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:15:52.469846 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:15:52.469854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:15:52.469854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:15:52.469860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:15:52.471563 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:52.471922 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:52.471924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:52.471925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:52.471979 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:15:52.472162 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:15:52.472163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:15:52.472163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:15:52.472311 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:15:52.472370 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:15:52.472538 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:15:52.472597 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260925 13:15:52.473186 16969 tablet_copy_service.cc:140] P ad631a2adb5a430a9f467cfc128a703b: Received BeginTabletCopySession request for tablet 71551d9484ac4206b034e0cf33f481d6 from peer bf98ce4c32594601a3ef73f751033caf ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48211)
I20260925 13:15:52.473278 16969 tablet_copy_service.cc:161] P ad631a2adb5a430a9f467cfc128a703b: Beginning new tablet copy session on tablet 71551d9484ac4206b034e0cf33f481d6 from peer bf98ce4c32594601a3ef73f751033caf at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48211: session id = bf98ce4c32594601a3ef73f751033caf-71551d9484ac4206b034e0cf33f481d6
I20260925 13:15:52.473879 16969 tablet_copy_source_session.cc:205] T 71551d9484ac4206b034e0cf33f481d6 P ad631a2adb5a430a9f467cfc128a703b: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 13:15:52.474820 17297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71551d9484ac4206b034e0cf33f481d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:15:52.477483 17297 tablet_copy_client.cc:791] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: tablet copy: Starting download of 0 data blocks...
I20260925 13:15:52.477654 17297 tablet_copy_client.cc:655] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: tablet copy: Starting download of 1 WAL segments...
I20260925 13:15:52.478559 17297 tablet_copy_client.cc:531] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 13:15:52.479319 17297 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Bootstrap starting.
I20260925 13:15:52.480713 17297 log.cc:824] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Log is configured to *not* fsync() on all Append() calls
I20260925 13:15:52.483186 17297 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:15:52.483419 17297 tablet_bootstrap.cc:492] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Bootstrap complete.
I20260925 13:15:52.483565 17297 ts_tablet_manager.cc:1397] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 13:15:52.484596 17297 raft_consensus.cc:348] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:52.484783 17297 raft_consensus.cc:729] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf98ce4c32594601a3ef73f751033caf, State: Initialized, Role: FOLLOWER
I20260925 13:15:52.484943 17297 consensus_queue.cc:260] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49845143cec411d9e6279854bcd96e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37207 } } peers { permanent_uuid: "bf98ce4c32594601a3ef73f751033caf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45607 } } peers { permanent_uuid: "ad631a2adb5a430a9f467cfc128a703b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36837 } }
I20260925 13:15:52.485216 17297 ts_tablet_manager.cc:1428] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 13:15:52.485514 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.485400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:41537)
0925 13:15:52.485460 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.485486 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 13:15:52.485514 16969 tablet_copy_service.cc:342] P ad631a2adb5a430a9f467cfc128a703b: Request end of tablet copy session bf98ce4c32594601a3ef73f751033caf-71551d9484ac4206b034e0cf33f481d6 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48211
I20260925 13:15:52.485603 16969 tablet_copy_service.cc:433] P ad631a2adb5a430a9f467cfc128a703b: ending tablet copy session bf98ce4c32594601a3ef73f751033caf-71551d9484ac4206b034e0cf33f481d6 on tablet 71551d9484ac4206b034e0cf33f481d6 with peer bf98ce4c32594601a3ef73f751033caf
I20260925 13:15:52.486074 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:52.485971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:48293)
0925 13:15:52.486029 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:52.486052 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 13:15:53.074990 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:53.074782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:53805)
0925 13:15:53.074900 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:53.074944 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260925 13:15:53.074970 17229 raft_consensus.cc:1203] T 71551d9484ac4206b034e0cf33f481d6 P bf98ce4c32594601a3ef73f751033caf [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 13:15:53.075655 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:53.075470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:51915)
0925 13:15:53.075575 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:53.075611 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":70,"threads_started":1}
I20260925 13:15:53.099088 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:53.098874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:53109)
0925 13:15:53.099007 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:53.099041 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260925 13:15:54.075771 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:54.075523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:50527)
0925 13:15:54.075655 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:54.075706 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260925 13:15:54.076318 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:54.076050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:60275)
0925 13:15:54.076223 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:54.076265 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260925 13:15:54.099737 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:54.099486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:49351)
0925 13:15:54.099602 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:54.099666 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260925 13:15:55.077118 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:55.076847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:46459)
0925 13:15:55.077030 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:55.077072 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260925 13:15:55.077296 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:55.076863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:47979)
0925 13:15:55.077172 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:55.077203 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260925 13:15:55.077545 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:55.077436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:49761)
0925 13:15:55.077490 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:55.077515 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 13:15:55.100436 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:55.100203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:45091)
0925 13:15:55.100333 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:55.100370 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 13:15:56.078231 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:56.077965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:49579)
0925 13:15:56.078153 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:56.078187 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260925 13:15:56.078226 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:56.077962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:42489)
0925 13:15:56.078132 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:56.078170 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260925 13:15:56.101128 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:56.100932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:41249)
0925 13:15:56.101048 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:56.101086 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260925 13:15:57.078938 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:57.078693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:39787)
0925 13:15:57.078823 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:57.078866 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260925 13:15:57.079272 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:57.079062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:37773)
0925 13:15:57.079193 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:57.079235 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260925 13:15:57.101892 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:57.101599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:43569)
0925 13:15:57.101807 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:57.101843 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":151,"threads_started":1}
I20260925 13:15:58.079641 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:58.079396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:54907)
0925 13:15:58.079538 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:58.079588 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260925 13:15:58.079916 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:58.079680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:44055)
0925 13:15:58.079842 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:58.079872 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20260925 13:15:58.102553 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:58.102352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:60535)
0925 13:15:58.102478 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:58.102512 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260925 13:15:59.080293 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:59.080078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:45967)
0925 13:15:59.080205 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:59.080250 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260925 13:15:59.080538 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:59.080313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:36107)
0925 13:15:59.080463 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:59.080505 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 13:15:59.103518 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:15:59.103295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:49019)
0925 13:15:59.103437 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:15:59.103469 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260925 13:16:00.080935 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:00.080722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:58361)
0925 13:16:00.080851 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:00.080890 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260925 13:16:00.081280 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:00.081079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:41315)
0925 13:16:00.081202 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:00.081242 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260925 13:16:00.104348 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:00.104028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:53561)
0925 13:16:00.104261 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:00.104301 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":79,"threads_started":1}
I20260925 13:16:01.081636 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:01.081402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:39253)
0925 13:16:01.081534 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:01.081575 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 13:16:01.082053 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:01.081862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:55849)
0925 13:16:01.081978 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:01.082016 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260925 13:16:01.105181 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:01.104962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:34385)
0925 13:16:01.105105 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:01.105135 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260925 13:16:02.082407 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:02.082157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:58355)
0925 13:16:02.082280 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:02.082318 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":109,"threads_started":1}
I20260925 13:16:02.082698 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:02.082479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:53577)
0925 13:16:02.082624 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:02.082658 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260925 13:16:02.105966 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:02.105727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:48273)
0925 13:16:02.105878 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:02.105917 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260925 13:16:03.083359 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:03.082995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:51799)
0925 13:16:03.083220 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:03.083288 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260925 13:16:03.083624 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:03.083355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:53339)
0925 13:16:03.083514 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:03.083563 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260925 13:16:03.106787 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:03.106518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:46583)
0925 13:16:03.106679 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:03.106718 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260925 13:16:04.084270 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:04.084035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:50117)
0925 13:16:04.084172 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:04.084220 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260925 13:16:04.084342 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:04.084122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:36455)
0925 13:16:04.084256 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:04.084300 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260925 13:16:04.107935 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:04.107548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:45869)
0925 13:16:04.107826 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:04.107880 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
I20260925 13:16:05.085011 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:05.084769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:41389)
0925 13:16:05.084924 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:05.084958 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260925 13:16:05.085050 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:05.084859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:35651)
0925 13:16:05.084971 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:05.085009 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260925 13:16:05.108875 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:05.108627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:40467)
0925 13:16:05.108773 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:05.108827 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 13:16:05.243022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 16842
I20260925 13:16:05.247970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17011
I20260925 13:16:05.252782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17146
2026-09-25T13:16:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58315 ms)
[----------] 30 tests from SecurityITest (233044 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:05 dist-test-slave-2d0s krb5kdc[17348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:05 dist-test-slave-2d0s krb5kdc[17348](info): set up 2 sockets
Sep 25 13:16:05 dist-test-slave-2d0s krb5kdc[17348](info): commencing operation
krb5kdc: starting...
W20260925 13:16:06.297528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:06.418066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:06.501019 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:06.501295 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:06.501331 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:06.502565 17361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:06.502648 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:06.502671 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:06.502694 17361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:06.502722 17361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:06.504215 17361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:06.504500 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:06.504741 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:06.509754 17366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:06.509820 17367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:06.509835 17361 server_base.cc:1034] running on GCE node
W20260925 13:16:06.509753 17369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:06.510231 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:06.510527 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:06.511670 17361 hybrid_clock.cc:648] HybridClock initialized: now 1790342166511649 us; error 37 us; skew 500 ppm
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:06.514250 17361 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:06.514707 17361 webserver.cc:460] Webserver started at http://127.0.98.62:38549/ using document root <none> and password file <none>
I20260925 13:16:06.514935 17361 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:06.514981 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:06.515136 17361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:06.516233 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "19abd1c64cc447569c0098ebeb0b391c"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "2583c9aeda89049d28d52354a63833a5"
server_key_iv: "de6fc756bae209c4b9c80a68ab936858"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.516607 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "19abd1c64cc447569c0098ebeb0b391c"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "2583c9aeda89049d28d52354a63833a5"
server_key_iv: "de6fc756bae209c4b9c80a68ab936858"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.517982 17361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:16:06.518704 17376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.518875 17361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:16:06.518967 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "19abd1c64cc447569c0098ebeb0b391c"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "2583c9aeda89049d28d52354a63833a5"
server_key_iv: "de6fc756bae209c4b9c80a68ab936858"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.519023 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:06.529716 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:06.531648 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:06.531762 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:06.535665 17361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38363
I20260925 13:16:06.535722 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38363 every 8 connection(s)
I20260925 13:16:06.536077 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:06.537288 17429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:06.539726 17429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c: Bootstrap starting.
I20260925 13:16:06.540335 17429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:06.540625 17429 log.cc:824] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:06.541347 17429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c: No bootstrap required, opened a new log
I20260925 13:16:06.543042 17429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abd1c64cc447569c0098ebeb0b391c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38363 } }
I20260925 13:16:06.543165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17361
I20260925 13:16:06.543205 17429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:06.543277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:06.543252 17429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abd1c64cc447569c0098ebeb0b391c, State: Initialized, Role: FOLLOWER
I20260925 13:16:06.543483 392 external_mini_cluster.cc:1428] Setting key 0fa9e384f0a32eb702ff097e8c12198f
I20260925 13:16:06.543453 17429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abd1c64cc447569c0098ebeb0b391c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38363 } }
I20260925 13:16:06.543538 17429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:06.543584 17429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:06.543637 17429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:06.544351 17429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abd1c64cc447569c0098ebeb0b391c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38363 } }
I20260925 13:16:06.544476 17429 leader_election.cc:302] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19abd1c64cc447569c0098ebeb0b391c; no voters:
I20260925 13:16:06.544674 17429 leader_election.cc:288] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:06.544728 17434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:06.544900 17434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [term 1 LEADER]: Becoming Leader. State: Replica: 19abd1c64cc447569c0098ebeb0b391c, State: Running, Role: LEADER
I20260925 13:16:06.545038 17434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abd1c64cc447569c0098ebeb0b391c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38363 } }
I20260925 13:16:06.545414 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19abd1c64cc447569c0098ebeb0b391c. Latest consensus state: current_term: 1 leader_uuid: "19abd1c64cc447569c0098ebeb0b391c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abd1c64cc447569c0098ebeb0b391c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38363 } } }
I20260925 13:16:06.545532 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:06.545421 17436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19abd1c64cc447569c0098ebeb0b391c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abd1c64cc447569c0098ebeb0b391c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38363 } } }
I20260925 13:16:06.545763 17436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:06.545791 17438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:06.546294 17438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:06.544920 17429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:06.547734 17438 catalog_manager.cc:1347] Generated new cluster ID: d6ed5fa44b234cc3936639a754db02f1
I20260925 13:16:06.547785 17438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:06.558456 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:06.544010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.0.98.62:38363)
0925 13:16:06.544585 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:06.544597 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:06.544636 (+ 39us) server_negotiation.cc:407] Connection header received
0925 13:16:06.544765 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:06.544773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:06.544832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:06.544976 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:06.545861 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.547302 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.549815 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.550496 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.550588 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:06.553764 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:06.553770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:06.553770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:06.553785 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:06.556608 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:06.557049 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:06.557053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:06.557054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:06.557186 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:06.557394 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:06.557395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:06.557395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:06.557787 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:06.557957 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:06.558053 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 13:16:06.558207 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":474,"threads_started":1}
I20260925 13:16:06.572079 17438 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:06.572876 17438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:06.578955 17438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c: Generated new TSK 0
I20260925 13:16:06.579196 17438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:16:06.604349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38363
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:06.688336 17455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:06.688578 17455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:06.688604 17455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:06.689889 17455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:06.689939 17455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:06.689972 17455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:16:06.691625 17455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:06.691912 17455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:06.692167 17455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:06.697275 17461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:06.697324 17460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:06.697427 17455 server_base.cc:1034] running on GCE node
W20260925 13:16:06.697363 17463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:06.697794 17455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:06.698037 17455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:06.699180 17455 hybrid_clock.cc:648] HybridClock initialized: now 1790342166699171 us; error 27 us; skew 500 ppm
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:06.701915 17455 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:16:06.702415 17455 webserver.cc:460] Webserver started at http://127.0.98.1:44783/ using document root <none> and password file <none>
I20260925 13:16:06.702665 17455 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:06.702714 17455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:06.702848 17455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:06.703958 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa072471a9484c7ea29f4e30620845b1"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "02d94d6974f7e07747548678f3732b47"
server_key_iv: "fe806313d0f5c86d1a287caea1624d77"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.704337 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa072471a9484c7ea29f4e30620845b1"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "02d94d6974f7e07747548678f3732b47"
server_key_iv: "fe806313d0f5c86d1a287caea1624d77"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.705770 17455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 13:16:06.706575 17470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.706755 17455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:06.706856 17455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "fa072471a9484c7ea29f4e30620845b1"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "02d94d6974f7e07747548678f3732b47"
server_key_iv: "fe806313d0f5c86d1a287caea1624d77"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.706944 17455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:06.714802 17455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:06.717298 17455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:06.717473 17455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:06.717787 17455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:06.718129 17455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:06.718191 17455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.718236 17455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:06.718262 17455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.724359 17455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43807
I20260925 13:16:06.724406 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43807 every 8 connection(s)
I20260925 13:16:06.724756 17455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:16:06.729489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17455
I20260925 13:16:06.729626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:16:06.729880 392 external_mini_cluster.cc:1428] Setting key 28f367435eddca5d6d7eac52d959016d
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:06.735957 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:06.725534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60331 (local address 127.0.98.62:38363)
0925 13:16:06.725603 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:06.725605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:06.725849 (+ 244us) server_negotiation.cc:407] Connection header received
0925 13:16:06.726874 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:06.726875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:06.726888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:06.726949 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:06.727658 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.728179 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.729269 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.729593 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.729714 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:06.732497 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:06.732499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:06.732499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:06.732510 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:06.734250 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:06.734794 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:06.734795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:06.734796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:06.734869 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:06.735125 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:06.735126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:06.735126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:06.735303 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:06.735385 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:06.735705 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 13:16:06.735811 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:16:06.735975 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:06.725509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38363 (local address 127.0.98.1:60331)
0925 13:16:06.725717 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:06.725763 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:06.726749 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:06.727042 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:06.727047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:06.727342 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:06.727552 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:06.727557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.728285 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.728287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:06.729162 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:06.729165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.729745 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.729747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:06.729939 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:06.730586 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:06.730599 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:06.732395 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:06.734352 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:06.734359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:06.734362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:06.734709 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:06.734941 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:06.734942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:06.734942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:06.735060 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:06.735393 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:06.735399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:06.735553 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 13:16:06.735652 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 13:16:06.735793 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":109,"threads_started":1}
I20260925 13:16:06.736574 17584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38363
I20260925 13:16:06.736706 17584 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:06.736909 17584 heartbeater.cc:511] Master 127.0.98.62:38363 requested a full tablet report, sending...
I20260925 13:16:06.737601 17393 ts_manager.cc:194] Registered new tserver with Master: fa072471a9484c7ea29f4e30620845b1 (127.0.98.1:43807)
I20260925 13:16:06.738619 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60331
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:16:06.756903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38363
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:06.840518 17589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:06.840765 17589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:06.840791 17589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:06.842115 17589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:06.842166 17589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:06.842197 17589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:16:06.843798 17589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:06.844080 17589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:06.844328 17589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:06.849846 17595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:06.849866 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:06.849846 17594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:06.850081 17589 server_base.cc:1034] running on GCE node
I20260925 13:16:06.850310 17589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:06.850535 17589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:06.851717 17589 hybrid_clock.cc:648] HybridClock initialized: now 1790342166851711 us; error 35 us; skew 500 ppm
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:06.854480 17589 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:16:06.854974 17589 webserver.cc:460] Webserver started at http://127.0.98.2:35315/ using document root <none> and password file <none>
I20260925 13:16:06.855218 17589 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:06.855293 17589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:06.855451 17589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:06.856642 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "77e139af23ae4848af84f3dffaf0ebfa"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "5c759899e6d261abaac7b8d9ab333872"
server_key_iv: "1171b1576c9d267073e10f20f528b4bb"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.857035 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "77e139af23ae4848af84f3dffaf0ebfa"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "5c759899e6d261abaac7b8d9ab333872"
server_key_iv: "1171b1576c9d267073e10f20f528b4bb"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.858573 17589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 13:16:06.859328 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.859505 17589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:06.859625 17589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "77e139af23ae4848af84f3dffaf0ebfa"
format_stamp: "Formatted at 2026-09-25 13:16:06 on dist-test-slave-2d0s"
server_key: "5c759899e6d261abaac7b8d9ab333872"
server_key_iv: "1171b1576c9d267073e10f20f528b4bb"
server_key_version: "encryptionkey@0"
I20260925 13:16:06.859704 17589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:06.870764 17589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:06.873135 17589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:06.873418 17589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:06.873737 17589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:06.874147 17589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:06.874202 17589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.874246 17589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:06.874272 17589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:06.880296 17589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46121
I20260925 13:16:06.880359 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46121 every 8 connection(s)
I20260925 13:16:06.880738 17589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:16:06.882272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17589
I20260925 13:16:06.882400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:16:06.882639 392 external_mini_cluster.cc:1428] Setting key 765fb2b3ccf84b8180ed92f381191258
Sep 25 13:16:06 dist-test-slave-2d0s krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:16:06.892347 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:06.881627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32985 (local address 127.0.98.62:38363)
0925 13:16:06.881766 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:06.881769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:06.881937 (+ 168us) server_negotiation.cc:407] Connection header received
0925 13:16:06.883359 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:06.883360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:06.883371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:06.883419 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:06.884162 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.884774 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.885908 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.886257 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.886334 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:06.888999 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:06.889000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:06.889001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:06.889010 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:06.890735 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:06.891204 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:06.891205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:06.891206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:06.891274 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:06.891591 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:06.891592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:06.891592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:06.891757 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:06.891841 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:06.892196 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 13:16:06.892273 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 13:16:06.892589 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:06.881534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38363 (local address 127.0.98.2:32985)
0925 13:16:06.881786 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:06.881838 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:06.883261 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:06.883500 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:06.883505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:06.883836 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:06.884059 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:06.884064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.884877 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.884878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:06.885823 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:06.885826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:06.886349 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:06.886350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:06.886458 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:06.887128 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:06.887146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:06.888877 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:06.890841 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:06.890849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:06.890852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:06.891117 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:06.891376 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:06.891377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:06.891378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:06.891511 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:06.891898 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:06.891904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:06.892061 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 13:16:06.892157 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 13:16:06.892311 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260925 13:16:06.893146 17718 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38363
I20260925 13:16:06.893263 17718 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:06.893491 17718 heartbeater.cc:511] Master 127.0.98.62:38363 requested a full tablet report, sending...
I20260925 13:16:06.893970 17393 ts_manager.cc:194] Registered new tserver with Master: 77e139af23ae4848af84f3dffaf0ebfa (127.0.98.2:46121)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 13:16:06.894353 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:16:06.908686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38363
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:06.993538 17723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:06.993782 17723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:06.993808 17723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:06.994941 17723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:06.994978 17723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:06.995004 17723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:16:06.996439 17723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:06.996651 17723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:06.996830 17723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:07.002048 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.002043 17731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.002043 17729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:07.002246 17723 server_base.cc:1034] running on GCE node
I20260925 13:16:07.002548 17723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:07.002800 17723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:07.003965 17723 hybrid_clock.cc:648] HybridClock initialized: now 1790342167003944 us; error 52 us; skew 500 ppm
Sep 25 13:16:07 dist-test-slave-2d0s krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:07.007413 17723 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:16:07.007953 17723 webserver.cc:460] Webserver started at http://127.0.98.3:44697/ using document root <none> and password file <none>
I20260925 13:16:07.008211 17723 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:07.008282 17723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:07.008446 17723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:07.009645 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9e104c5fc9f429c9dd9b39d79edde0f"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "382eb7b2739743fec248295dffffbcbd"
server_key_iv: "c3b1a5338df70f6913a1e687a7943d4a"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.010140 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9e104c5fc9f429c9dd9b39d79edde0f"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "382eb7b2739743fec248295dffffbcbd"
server_key_iv: "c3b1a5338df70f6913a1e687a7943d4a"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.011608 17723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:16:07.012503 17738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.012727 17723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:16:07.012837 17723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "e9e104c5fc9f429c9dd9b39d79edde0f"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "382eb7b2739743fec248295dffffbcbd"
server_key_iv: "c3b1a5338df70f6913a1e687a7943d4a"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.012933 17723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:07.032338 17723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:07.034865 17723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:07.035068 17723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:07.035341 17723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:07.035697 17723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:07.035760 17723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.035804 17723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:07.035832 17723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.041975 17723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45841
I20260925 13:16:07.042023 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45841 every 8 connection(s)
I20260925 13:16:07.042377 17723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:16:07.044220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17723
I20260925 13:16:07.044369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:16:07.044579 392 external_mini_cluster.cc:1428] Setting key 12049d9859bd69d4e8620377d5d59697
Sep 25 13:16:07 dist-test-slave-2d0s krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:07.052984 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.043179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44003 (local address 127.0.98.62:38363)
0925 13:16:07.043232 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.043234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.043453 (+ 219us) server_negotiation.cc:407] Connection header received
0925 13:16:07.044675 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.044677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.044693 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.044776 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.045720 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.046368 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.047564 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.047767 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.047822 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.050255 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.050257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.050258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:07.050265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.051585 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:07.051966 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:07.051967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:07.051968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:07.052021 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:07.052328 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:07.052330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:07.052330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:07.052481 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.052550 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.052815 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.052902 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:07.053143 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.043163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38363 (local address 127.0.98.3:44003)
0925 13:16:07.043315 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.043354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.044514 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.044887 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.044892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.045308 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:07.045580 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.045587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.046454 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.046455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.047416 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.047418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.047848 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.047849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.047967 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.048544 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:07.048553 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:07.050166 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:07.051660 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:07.051667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:07.051669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:07.051900 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:07.052103 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:07.052104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:07.052104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:07.052224 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:07.052554 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:07.052559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:07.052691 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.052778 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.052916 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 13:16:07.053738 17852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38363
I20260925 13:16:07.053843 17852 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:07.054008 17852 heartbeater.cc:511] Master 127.0.98.62:38363 requested a full tablet report, sending...
I20260925 13:16:07.054426 17393 ts_manager.cc:194] Registered new tserver with Master: e9e104c5fc9f429c9dd9b39d79edde0f (127.0.98.3:45841)
I20260925 13:16:07.054818 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44003
I20260925 13:16:07.056536 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:16:07.064698 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.058049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.98.62:38363)
0925 13:16:07.058109 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.058111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.058233 (+ 122us) server_negotiation.cc:407] Connection header received
0925 13:16:07.058279 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.058280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.058291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.058352 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.058968 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.059486 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.060277 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.060457 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.060508 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.061833 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.061834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.061835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:07.061841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.063436 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:07.063858 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:07.063859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:07.063859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:07.063912 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:07.064096 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:07.064096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:07.064096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:07.064256 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.064336 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.064564 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.064628 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:16:07.066479 17393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:16:07.068173 17393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:16:07.074793 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.070463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45841 (local address 127.0.0.1:40190)
0925 13:16:07.070794 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.070842 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.070913 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.071230 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.071231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.071233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.071289 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.071291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.072323 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.072324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.073654 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.073683 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.074432 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.074433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.074514 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.074537 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.074637 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.074701 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":35,"thread_start_us":83,"threads_started":1}
I20260925 13:16:07.074810 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.070598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.0.98.3:45841)
0925 13:16:07.071069 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.071074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.071084 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:16:07.071122 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.071124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.071140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.071218 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.071379 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.072218 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.073755 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.074341 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.074432 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.074500 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.074689 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.074732 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":97,"threads_started":1}
I20260925 13:16:07.075001 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.070472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46121 (local address 127.0.0.1:49558)
0925 13:16:07.070798 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.070841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.070921 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.071159 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.071164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.071166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.071250 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.071252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.072626 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.072627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.073737 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.073739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.074732 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.074734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.074813 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.074819 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.074866 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.074912 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":69,"threads_started":1}
I20260925 13:16:07.075178 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.070917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.98.1:43807)
0925 13:16:07.071395 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.071399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.071408 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:07.071449 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.071451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.071467 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.071537 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.071752 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.072771 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.073995 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.074697 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.074759 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.074812 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.075047 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.075101 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":81,"threads_started":1}
I20260925 13:16:07.075270 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.070579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.0.98.2:46121)
0925 13:16:07.070773 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.070780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.070914 (+ 134us) server_negotiation.cc:407] Connection header received
0925 13:16:07.071012 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.071015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.071037 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.071146 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.071325 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.072507 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.073829 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.074561 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.074635 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.074694 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.075131 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.075179 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260925 13:16:07.075335 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.070710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43807 (local address 127.0.0.1:60118)
0925 13:16:07.071018 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.071026 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.071064 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.071621 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.071622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.071624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.071667 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.071668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.072871 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.072872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.073904 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.073907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.074842 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.074843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.074899 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.074902 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.074959 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.074992 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":49,"threads_started":1}
I20260925 13:16:07.075889 17786 tablet_service.cc:1467] Processing CreateTablet for tablet 3416454898d34181aad3b7db5673fdc6 (DEFAULT_TABLE table=test-table [id=07d9f9d5fdb9459196b4b880ad075f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:07.075935 17652 tablet_service.cc:1467] Processing CreateTablet for tablet 3416454898d34181aad3b7db5673fdc6 (DEFAULT_TABLE table=test-table [id=07d9f9d5fdb9459196b4b880ad075f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:07.075944 17518 tablet_service.cc:1467] Processing CreateTablet for tablet 3416454898d34181aad3b7db5673fdc6 (DEFAULT_TABLE table=test-table [id=07d9f9d5fdb9459196b4b880ad075f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:07.076267 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416454898d34181aad3b7db5673fdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.076267 17518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416454898d34181aad3b7db5673fdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.076268 17786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416454898d34181aad3b7db5673fdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.079703 17866 tablet_bootstrap.cc:492] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Bootstrap starting.
I20260925 13:16:07.080533 17868 tablet_bootstrap.cc:492] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Bootstrap starting.
I20260925 13:16:07.080669 17866 tablet_bootstrap.cc:654] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.080734 17867 tablet_bootstrap.cc:492] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1: Bootstrap starting.
I20260925 13:16:07.081290 17868 tablet_bootstrap.cc:654] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.081279 17867 tablet_bootstrap.cc:654] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.081296 17866 log.cc:824] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.081935 17868 log.cc:824] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.081955 17867 log.cc:824] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.082226 17866 tablet_bootstrap.cc:492] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: No bootstrap required, opened a new log
I20260925 13:16:07.082326 17866 ts_tablet_manager.cc:1397] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:16:07.082716 17867 tablet_bootstrap.cc:492] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1: No bootstrap required, opened a new log
I20260925 13:16:07.082773 17867 ts_tablet_manager.cc:1397] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:07.084585 17866 raft_consensus.cc:348] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.084776 17866 raft_consensus.cc:374] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.084820 17866 raft_consensus.cc:729] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e104c5fc9f429c9dd9b39d79edde0f, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.084789 17867 raft_consensus.cc:348] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.084933 17867 raft_consensus.cc:374] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.084964 17867 raft_consensus.cc:729] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa072471a9484c7ea29f4e30620845b1, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.084949 17866 consensus_queue.cc:260] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.085065 17867 consensus_queue.cc:260] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.085295 17852 heartbeater.cc:503] Master 127.0.98.62:38363 was elected leader, sending a full tablet report...
I20260925 13:16:07.085328 17867 ts_tablet_manager.cc:1428] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:07.085350 17584 heartbeater.cc:503] Master 127.0.98.62:38363 was elected leader, sending a full tablet report...
I20260925 13:16:07.085733 17866 ts_tablet_manager.cc:1428] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:07.085937 17868 tablet_bootstrap.cc:492] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: No bootstrap required, opened a new log
I20260925 13:16:07.086042 17868 ts_tablet_manager.cc:1397] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 13:16:07.087751 17868 raft_consensus.cc:348] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.087929 17868 raft_consensus.cc:374] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.087978 17868 raft_consensus.cc:729] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e139af23ae4848af84f3dffaf0ebfa, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.088110 17868 consensus_queue.cc:260] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.088397 17868 ts_tablet_manager.cc:1428] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:07.088455 17718 heartbeater.cc:503] Master 127.0.98.62:38363 was elected leader, sending a full tablet report...
W20260925 13:16:07.131402 17719 tablet.cc:2369] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:07.152604 17874 raft_consensus.cc:482] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:16:07.152756 17874 raft_consensus.cc:504] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.153174 17874 leader_election.cc:288] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9e104c5fc9f429c9dd9b39d79edde0f (127.0.98.3:45841), fa072471a9484c7ea29f4e30620845b1 (127.0.98.1:43807)
I20260925 13:16:07.156255 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.153493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42343 (local address 127.0.98.3:45841)
0925 13:16:07.153584 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.153587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.153599 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:16:07.153635 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.153636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.153649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.153747 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.153945 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.154438 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.155449 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.155840 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.155941 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.155954 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.156122 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.156169 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:16:07.156283 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.153405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45841 (local address 127.0.98.2:42343)
0925 13:16:07.153485 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.153501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.153574 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.153772 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.153774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.153776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.153864 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.153866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.154534 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.154536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.155327 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.155329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.155939 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.155940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.156032 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.156036 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.156115 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.156165 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 13:16:07.156287 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.153758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47857 (local address 127.0.98.1:43807)
0925 13:16:07.153809 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.153811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.153823 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:16:07.153905 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.153906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.153919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.153990 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.154164 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.154626 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.155574 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.155941 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.156021 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.156030 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.156176 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.156217 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:16:07.156312 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.153550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43807 (local address 127.0.98.2:47857)
0925 13:16:07.153776 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.153786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.153837 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.153992 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.153993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.153995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.154082 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.154083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.154711 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.154712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.155478 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.155480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.156021 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.156022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.156101 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.156104 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.156171 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.156211 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":161,"threads_started":1}
I20260925 13:16:07.156690 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416454898d34181aad3b7db5673fdc6" candidate_uuid: "77e139af23ae4848af84f3dffaf0ebfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" is_pre_election: true
I20260925 13:16:07.156939 17806 raft_consensus.cc:2454] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e139af23ae4848af84f3dffaf0ebfa in term 0.
I20260925 13:16:07.156901 17538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416454898d34181aad3b7db5673fdc6" candidate_uuid: "77e139af23ae4848af84f3dffaf0ebfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa072471a9484c7ea29f4e30620845b1" is_pre_election: true
I20260925 13:16:07.157070 17538 raft_consensus.cc:2454] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e139af23ae4848af84f3dffaf0ebfa in term 0.
I20260925 13:16:07.157219 17607 leader_election.cc:302] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e139af23ae4848af84f3dffaf0ebfa, e9e104c5fc9f429c9dd9b39d79edde0f; no voters:
I20260925 13:16:07.157356 17874 raft_consensus.cc:2781] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:16:07.157423 17874 raft_consensus.cc:482] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:16:07.157454 17874 raft_consensus.cc:3037] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:07.158401 17874 raft_consensus.cc:504] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.158558 17874 leader_election.cc:288] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [CANDIDATE]: Term 1 election: Requested vote from peers e9e104c5fc9f429c9dd9b39d79edde0f (127.0.98.3:45841), fa072471a9484c7ea29f4e30620845b1 (127.0.98.1:43807)
I20260925 13:16:07.158769 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416454898d34181aad3b7db5673fdc6" candidate_uuid: "77e139af23ae4848af84f3dffaf0ebfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f"
I20260925 13:16:07.158777 17538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416454898d34181aad3b7db5673fdc6" candidate_uuid: "77e139af23ae4848af84f3dffaf0ebfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa072471a9484c7ea29f4e30620845b1"
I20260925 13:16:07.158874 17806 raft_consensus.cc:3037] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:07.158867 17538 raft_consensus.cc:3037] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:07.159870 17806 raft_consensus.cc:2454] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e139af23ae4848af84f3dffaf0ebfa in term 1.
I20260925 13:16:07.159864 17538 raft_consensus.cc:2454] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e139af23ae4848af84f3dffaf0ebfa in term 1.
I20260925 13:16:07.160091 17607 leader_election.cc:302] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e139af23ae4848af84f3dffaf0ebfa, fa072471a9484c7ea29f4e30620845b1; no voters:
I20260925 13:16:07.160207 17874 raft_consensus.cc:2781] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:07.160426 17874 raft_consensus.cc:686] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 1 LEADER]: Becoming Leader. State: Replica: 77e139af23ae4848af84f3dffaf0ebfa, State: Running, Role: LEADER
I20260925 13:16:07.160552 17874 consensus_queue.cc:237] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260925 13:16:07.161516 17393 catalog_manager.cc:5495] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e139af23ae4848af84f3dffaf0ebfa (127.0.98.2). New cstate: current_term: 1 leader_uuid: "77e139af23ae4848af84f3dffaf0ebfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e139af23ae4848af84f3dffaf0ebfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } health_report { overall_health: UNKNOWN } } }
I20260925 13:16:07.172976 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.170448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.0.98.2:46121)
0925 13:16:07.170493 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.170495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.170511 (+ 16us) server_negotiation.cc:407] Connection header received
0925 13:16:07.170557 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.170558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.170580 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.170681 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:07.170811 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.171301 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.172109 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.172320 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.172397 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.172589 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:07.172704 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:07.172807 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.172825 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.172848 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:16:07.174961 17538 raft_consensus.cc:1261] T 3416454898d34181aad3b7db5673fdc6 P fa072471a9484c7ea29f4e30620845b1 [term 1 FOLLOWER]: Refusing update from remote peer 77e139af23ae4848af84f3dffaf0ebfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:07.174961 17806 raft_consensus.cc:1261] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 1 FOLLOWER]: Refusing update from remote peer 77e139af23ae4848af84f3dffaf0ebfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:07.175335 17876 consensus_queue.cc:1035] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e104c5fc9f429c9dd9b39d79edde0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:07.175505 17874 consensus_queue.cc:1035] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa072471a9484c7ea29f4e30620845b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:07.178166 17880 mvcc.cc:205] Tried to move back new op lower bound from 7333241516745707520 to 7333241516690493440. Current Snapshot: MvccSnapshot[applied={T|T < 7333241516745707520}]
I20260925 13:16:07.181411 17392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:07.181522 17392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:07.182466 17392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19abd1c64cc447569c0098ebeb0b391c: Sending DeleteTablet for 3 replicas of tablet 3416454898d34181aad3b7db5673fdc6
I20260925 13:16:07.182819 17786 tablet_service.cc:1514] Processing DeleteTablet for tablet 3416454898d34181aad3b7db5673fdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40190
I20260925 13:16:07.182886 17652 tablet_service.cc:1514] Processing DeleteTablet for tablet 3416454898d34181aad3b7db5673fdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49558
I20260925 13:16:07.183065 17889 tablet_replica.cc:330] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: stopping tablet replica
I20260925 13:16:07.183135 17518 tablet_service.cc:1514] Processing DeleteTablet for tablet 3416454898d34181aad3b7db5673fdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60118
I20260925 13:16:07.183203 17889 raft_consensus.cc:2229] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:07.183326 17889 raft_consensus.cc:2258] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:07.183529 17890 tablet_replica.cc:330] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: stopping tablet replica
I20260925 13:16:07.183625 17890 raft_consensus.cc:2229] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:16:07.183660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17455
I20260925 13:16:07.183763 17890 raft_consensus.cc:2258] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:07.184178 17889 ts_tablet_manager.cc:1905] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:07.184552 17890 ts_tablet_manager.cc:1905] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:07.186220 17889 ts_tablet_manager.cc:1918] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:07.186383 17889 log.cc:1196] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/3416454898d34181aad3b7db5673fdc6
I20260925 13:16:07.186378 17890 ts_tablet_manager.cc:1918] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:07.186444 17890 log.cc:1196] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/3416454898d34181aad3b7db5673fdc6
I20260925 13:16:07.186755 17890 ts_tablet_manager.cc:1939] T 3416454898d34181aad3b7db5673fdc6 P 77e139af23ae4848af84f3dffaf0ebfa: Deleting consensus metadata
I20260925 13:16:07.186861 17889 ts_tablet_manager.cc:1939] T 3416454898d34181aad3b7db5673fdc6 P e9e104c5fc9f429c9dd9b39d79edde0f: Deleting consensus metadata
I20260925 13:16:07.187302 17379 catalog_manager.cc:4841] TS e9e104c5fc9f429c9dd9b39d79edde0f (127.0.98.3:45841): tablet 3416454898d34181aad3b7db5673fdc6 (table test-table [id=07d9f9d5fdb9459196b4b880ad075f3b]) successfully deleted
I20260925 13:16:07.187562 17377 catalog_manager.cc:4841] TS 77e139af23ae4848af84f3dffaf0ebfa (127.0.98.2:46121): tablet 3416454898d34181aad3b7db5673fdc6 (table test-table [id=07d9f9d5fdb9459196b4b880ad075f3b]) successfully deleted
W20260925 13:16:07.192631 17379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43807 (error 108)
I20260925 13:16:07.192759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17589
I20260925 13:16:07.194190 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.193108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43807 (local address 127.0.0.1:60122)
0925 13:16:07.194078 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.194137 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":936}
W20260925 13:16:07.194444 17379 catalog_manager.cc:4584] TS fa072471a9484c7ea29f4e30620845b1 (127.0.98.1:43807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3416454898d34181aad3b7db5673fdc6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43807: connect: Connection refused (error 111)
I20260925 13:16:07.198119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17723
I20260925 13:16:07.202690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17361
2026-09-25T13:16:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1949 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T13:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:07Z Disabled control of system clock
I20260925 13:16:07.230786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46019
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:07.315891 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:07.316135 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:07.316159 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:07.317310 17896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:07.317348 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:07.317360 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:07.317373 17896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:07.317385 17896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:07.318817 17896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:07.319049 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:07.319308 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:07.324150 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.324121 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.324132 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:07.324477 17896 server_base.cc:1034] running on GCE node
I20260925 13:16:07.324746 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:07.325058 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:07.326203 17896 hybrid_clock.cc:648] HybridClock initialized: now 1790342167326181 us; error 39 us; skew 500 ppm
I20260925 13:16:07.327570 17896 webserver.cc:460] Webserver started at http://127.0.98.62:45071/ using document root <none> and password file <none>
I20260925 13:16:07.327880 17896 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:07.327951 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:07.328119 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:07.329239 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "1d7dc206139b4ac7bdea8e49d83288f2"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "a06bf7c4914c7ccdc8f8436f47a0b684"
server_key_iv: "3dcfcbb70c3d4f80049752488586c738"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.329615 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d7dc206139b4ac7bdea8e49d83288f2"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "a06bf7c4914c7ccdc8f8436f47a0b684"
server_key_iv: "3dcfcbb70c3d4f80049752488586c738"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.331002 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:07.331912 17910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.332145 17896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:07.332253 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "1d7dc206139b4ac7bdea8e49d83288f2"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "a06bf7c4914c7ccdc8f8436f47a0b684"
server_key_iv: "3dcfcbb70c3d4f80049752488586c738"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.332343 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:07.342219 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:07.342581 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:07.342751 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:07.347067 17896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46019
I20260925 13:16:07.347102 17962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46019 every 8 connection(s)
I20260925 13:16:07.347466 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:07.348531 17963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.351015 17963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: Bootstrap starting.
I20260925 13:16:07.351579 17963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.351821 17963 log.cc:824] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.352537 17963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: No bootstrap required, opened a new log
I20260925 13:16:07.354221 17963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260925 13:16:07.354365 17963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.354388 17963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7dc206139b4ac7bdea8e49d83288f2, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.354544 17963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260925 13:16:07.354625 17963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:07.354648 17963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:07.354678 17963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:07.355180 17963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260925 13:16:07.355265 17963 leader_election.cc:302] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d7dc206139b4ac7bdea8e49d83288f2; no voters:
I20260925 13:16:07.355331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17896
I20260925 13:16:07.355460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:07.355502 17963 leader_election.cc:288] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:07.355566 17966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:07.355731 392 external_mini_cluster.cc:1428] Setting key 8a41ddeebb6656e7e2d269456d8a9cae
I20260925 13:16:07.355751 17966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [term 1 LEADER]: Becoming Leader. State: Replica: 1d7dc206139b4ac7bdea8e49d83288f2, State: Running, Role: LEADER
I20260925 13:16:07.355777 17963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:07.355883 17966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260925 13:16:07.356508 17968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d7dc206139b4ac7bdea8e49d83288f2. Latest consensus state: current_term: 1 leader_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } } }
I20260925 13:16:07.356504 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7dc206139b4ac7bdea8e49d83288f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } } }
I20260925 13:16:07.356624 17968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:07.356642 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2 [sys.catalog]: This master's current role is: LEADER
W20260925 13:16:07.357827 17983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:07.357900 17983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:07.357972 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:07.358139 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:07.359851 17977 catalog_manager.cc:1347] Generated new cluster ID: 7204f6fd645144b0bce26c60f09d42ea
I20260925 13:16:07.359913 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:07.362143 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.356313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.0.98.62:46019)
0925 13:16:07.356643 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.356653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.356691 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:16:07.356819 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.356826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.356904 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.357126 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.357360 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.358761 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.359836 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.360348 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.360503 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.361290 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.361294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.361294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:07.361514 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.361549 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:07.361561 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.361647 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.361883 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.361979 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260925 13:16:07.376181 17977 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:07.376824 17977 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:07.384076 17977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: Generated new TSK 0
I20260925 13:16:07.384347 17977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:07.387650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46019
--builtin_ntp_servers=127.0.98.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:07.487757 17987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:07.488020 17987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:07.488062 17987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:07.489284 17987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:07.489367 17987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:07.489414 17987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:16:07.490934 17987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:07.491235 17987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:07.491567 17987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:07.497819 17992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.497944 17993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.497855 17995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:07.498068 17987 server_base.cc:1034] running on GCE node
I20260925 13:16:07.498287 17987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:07.498575 17987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:07.499745 17987 hybrid_clock.cc:648] HybridClock initialized: now 1790342167499724 us; error 39 us; skew 500 ppm
I20260925 13:16:07.501219 17987 webserver.cc:460] Webserver started at http://127.0.98.1:46035/ using document root <none> and password file <none>
I20260925 13:16:07.501508 17987 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:07.501566 17987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:07.501729 17987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:07.503159 17987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "a0cbf2f4c5be4e53981f52556bb1a77b"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "f2911c1776e7d6384919d4272cede1a6"
server_key_iv: "7a29c2c06564f734e83802ff847fb8c4"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.503571 17987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a0cbf2f4c5be4e53981f52556bb1a77b"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "f2911c1776e7d6384919d4272cede1a6"
server_key_iv: "7a29c2c06564f734e83802ff847fb8c4"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.505654 17987 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 13:16:07.506632 18001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.506886 17987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:07.506987 17987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "a0cbf2f4c5be4e53981f52556bb1a77b"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "f2911c1776e7d6384919d4272cede1a6"
server_key_iv: "7a29c2c06564f734e83802ff847fb8c4"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.507076 17987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:07.525560 17987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:07.525985 17987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:07.526114 17987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:07.526312 17987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:07.526685 17987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:07.526721 17987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.526770 17987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:07.526800 17987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.533262 17987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35077
I20260925 13:16:07.533342 18114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35077 every 8 connection(s)
I20260925 13:16:07.533777 17987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:16:07.539122 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.534570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46067 (local address 127.0.98.62:46019)
0925 13:16:07.534662 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.534664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.534897 (+ 233us) server_negotiation.cc:407] Connection header received
0925 13:16:07.535400 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.535401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.535414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.535467 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.536222 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.536771 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.537807 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.538011 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.538066 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.538856 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.538857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.538857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:07.538862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.538876 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:07.538895 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.538940 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.539028 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.539057 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 13:16:07.539361 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.534525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46019 (local address 127.0.98.1:46067)
0925 13:16:07.534764 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.534801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.535313 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.535528 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.535531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.535540 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:07.536140 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.536147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.536857 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.536858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.537726 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.537729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.538085 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.538086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.538202 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.538721 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:07.538721 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:07.538750 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:07.538775 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:07.538953 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:07.538957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:07.538960 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.539020 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.539106 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260925 13:16:07.539925 18115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46019
I20260925 13:16:07.540027 18115 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:07.540177 18115 heartbeater.cc:511] Master 127.0.98.62:46019 requested a full tablet report, sending...
I20260925 13:16:07.540671 17927 ts_manager.cc:194] Registered new tserver with Master: a0cbf2f4c5be4e53981f52556bb1a77b (127.0.98.1:35077)
I20260925 13:16:07.541266 17927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46067
I20260925 13:16:07.543841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 17987
I20260925 13:16:07.543946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:16:07.544117 392 external_mini_cluster.cc:1428] Setting key d8bb363d5ccdfc126333fe0d06c7cb8c
I20260925 13:16:07.545077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46019
--builtin_ntp_servers=127.0.98.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:07.630059 18118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:07.630300 18118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:07.630335 18118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:07.631500 18118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:07.631549 18118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:07.631577 18118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:16:07.632895 18118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:07.633121 18118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:07.633420 18118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:07.638837 18123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.638863 18124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.638839 18126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:07.639581 18118 server_base.cc:1034] running on GCE node
I20260925 13:16:07.639849 18118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:07.640127 18118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:07.641278 18118 hybrid_clock.cc:648] HybridClock initialized: now 1790342167641269 us; error 28 us; skew 500 ppm
I20260925 13:16:07.642791 18118 webserver.cc:460] Webserver started at http://127.0.98.2:35307/ using document root <none> and password file <none>
I20260925 13:16:07.643083 18118 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:07.643136 18118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:07.643318 18118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:07.644484 18118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "75607f90588641c5b4e250282373dd6b"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "4c4f209aca9393e904e43d13bfac858d"
server_key_iv: "b5bb5e5dbe505b4c3d979db8a420fc98"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.644869 18118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "75607f90588641c5b4e250282373dd6b"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "4c4f209aca9393e904e43d13bfac858d"
server_key_iv: "b5bb5e5dbe505b4c3d979db8a420fc98"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.646371 18118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:07.647185 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.647377 18118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:07.647470 18118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "75607f90588641c5b4e250282373dd6b"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "4c4f209aca9393e904e43d13bfac858d"
server_key_iv: "b5bb5e5dbe505b4c3d979db8a420fc98"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.647547 18118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:07.665251 18118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:07.665622 18118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:07.665858 18118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:07.666128 18118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:07.666496 18118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:07.666538 18118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.666563 18118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:07.666612 18118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.672771 18118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39391
I20260925 13:16:07.672817 18245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39391 every 8 connection(s)
I20260925 13:16:07.673194 18118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:16:07.678802 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.674047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44609 (local address 127.0.98.62:46019)
0925 13:16:07.674125 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.674127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.674378 (+ 251us) server_negotiation.cc:407] Connection header received
0925 13:16:07.674913 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.674914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.674926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.675003 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.675758 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.676291 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.677299 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.677490 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.677542 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.678552 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.678553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.678553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:07.678558 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.678573 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:07.678577 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.678622 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.678698 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.678725 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:16:07.679061 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.674053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46019 (local address 127.0.98.2:44609)
0925 13:16:07.674244 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.674280 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.674779 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.675062 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.675065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.675074 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:07.675663 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.675667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.676380 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.676381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.677220 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.677222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.677568 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.677569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.677753 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.678399 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:07.678400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:07.678432 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:07.678459 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:07.678623 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:07.678625 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:07.678628 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.678683 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.678769 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260925 13:16:07.679736 18246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46019
I20260925 13:16:07.679837 18246 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:07.679999 18246 heartbeater.cc:511] Master 127.0.98.62:46019 requested a full tablet report, sending...
I20260925 13:16:07.680459 17927 ts_manager.cc:194] Registered new tserver with Master: 75607f90588641c5b4e250282373dd6b (127.0.98.2:39391)
I20260925 13:16:07.680509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18118
I20260925 13:16:07.680636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:16:07.680876 392 external_mini_cluster.cc:1428] Setting key 66650ab0e0b9b9c32ece17399586afa7
I20260925 13:16:07.680951 17927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44609
I20260925 13:16:07.681958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46019
--builtin_ntp_servers=127.0.98.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:07.766831 18249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:07.767069 18249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:07.767094 18249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:07.768343 18249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:07.768393 18249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:07.768420 18249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:16:07.769878 18249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:07.770170 18249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:07.770529 18249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:07.775722 18257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.775722 18255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:07.775728 18254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:07.776042 18249 server_base.cc:1034] running on GCE node
I20260925 13:16:07.776321 18249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:07.776648 18249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:07.777760 18249 hybrid_clock.cc:648] HybridClock initialized: now 1790342167777742 us; error 32 us; skew 500 ppm
I20260925 13:16:07.779460 18249 webserver.cc:460] Webserver started at http://127.0.98.3:41705/ using document root <none> and password file <none>
I20260925 13:16:07.779759 18249 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:07.779811 18249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:07.779958 18249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:07.781155 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "046fae86ecf65571b30fc11681a06c5f"
server_key_iv: "1572d9d8678eafffebc7d9fff07a6634"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.781553 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "046fae86ecf65571b30fc11681a06c5f"
server_key_iv: "1572d9d8678eafffebc7d9fff07a6634"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.782992 18249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:07.783869 18263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.784099 18249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:07.784225 18249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0"
format_stamp: "Formatted at 2026-09-25 13:16:07 on dist-test-slave-2d0s"
server_key: "046fae86ecf65571b30fc11681a06c5f"
server_key_iv: "1572d9d8678eafffebc7d9fff07a6634"
server_key_version: "encryptionkey@0"
I20260925 13:16:07.784293 18249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:07.799675 18249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:07.800051 18249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:07.800225 18249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:07.800509 18249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:07.800899 18249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:07.800963 18249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.801014 18249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:07.801043 18249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:07.807631 18249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39747
I20260925 13:16:07.807682 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39747 every 8 connection(s)
I20260925 13:16:07.808115 18249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:16:07.814232 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.809031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53359 (local address 127.0.98.62:46019)
0925 13:16:07.809116 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.809119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.809396 (+ 277us) server_negotiation.cc:407] Connection header received
0925 13:16:07.810039 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.810040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.810055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.810113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.810907 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.811463 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.812705 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.812891 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.812955 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.813941 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.813942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.813942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:07.813947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.813962 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:07.813965 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.814006 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.814139 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.814164 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 13:16:07.814465 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.808993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46019 (local address 127.0.98.3:53359)
0925 13:16:07.809260 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.809299 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.809941 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.810197 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.810201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.810210 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:07.810754 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.810759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.811576 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.811577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.812618 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.812621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.812980 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.812981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.813100 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.813788 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:07.813789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:07.813826 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:07.813853 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:07.814043 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:07.814050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:07.814054 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.814131 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.814224 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":126,"threads_started":1}
I20260925 13:16:07.815063 18377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46019
I20260925 13:16:07.815161 18377 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:07.815316 18377 heartbeater.cc:511] Master 127.0.98.62:46019 requested a full tablet report, sending...
I20260925 13:16:07.815701 17927 ts_manager.cc:194] Registered new tserver with Master: b4bb9cabbe8344c69c864bbf6fbcf5a0 (127.0.98.3:39747)
I20260925 13:16:07.816044 17927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53359
I20260925 13:16:07.817279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18249
I20260925 13:16:07.817394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:16:07.817588 392 external_mini_cluster.cc:1428] Setting key 2e4584acc6dc7f5b9925eb3cab8a4675
I20260925 13:16:07.818608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:16:07.822949 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.820006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.0.98.62:46019)
0925 13:16:07.820054 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.820056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.820209 (+ 153us) server_negotiation.cc:407] Connection header received
0925 13:16:07.820235 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.820236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.820246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.820301 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:07.820443 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.820949 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.821695 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.821914 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.821977 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.822679 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:07.822681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:07.822681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:07.822685 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:07.822701 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:07.822705 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:07.822758 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.822811 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.822834 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:07.824790 17927 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:16:07.826944 17927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:16:07.834029 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.829421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.0.98.1:35077)
0925 13:16:07.829587 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.829592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.830047 (+ 455us) server_negotiation.cc:407] Connection header received
0925 13:16:07.830160 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.830162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.830180 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.830255 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.830523 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.831708 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.832857 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.833513 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.833590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.833717 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.833880 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.833923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260925 13:16:07.834049 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.829328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35077 (local address 127.0.0.1:54578)
0925 13:16:07.829936 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.829969 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.830059 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.830344 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.830347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.830349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.830428 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.830431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.831808 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.831808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.832759 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.832762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.833611 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.833612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.833748 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.833769 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.833853 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.833920 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":144,"threads_started":1}
I20260925 13:16:07.834254 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.829622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.0.98.2:39391)
0925 13:16:07.829863 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.829868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.830225 (+ 357us) server_negotiation.cc:407] Connection header received
0925 13:16:07.830318 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.830321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.830339 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.830453 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.830587 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.831395 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.833020 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.833670 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.833763 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.833869 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.834095 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.834145 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":135,"threads_started":1}
I20260925 13:16:07.834504 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.829331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39391 (local address 127.0.0.1:56934)
0925 13:16:07.830165 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.830174 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.830223 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.830451 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.830452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.830455 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.830509 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.830511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.831505 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.831506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.832909 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.832912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.833905 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.833906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.833958 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.833961 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.834015 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.834052 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":89,"threads_started":1}
I20260925 13:16:07.834970 18049 tablet_service.cc:1467] Processing CreateTablet for tablet a9bcf8b7bff14b6f943ddabd57016006 (DEFAULT_TABLE table=test-table [id=987cbfa1ca3f4ac9894f86a7dbb47ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:07.835363 18049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bcf8b7bff14b6f943ddabd57016006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.835345 18180 tablet_service.cc:1467] Processing CreateTablet for tablet a9bcf8b7bff14b6f943ddabd57016006 (DEFAULT_TABLE table=test-table [id=987cbfa1ca3f4ac9894f86a7dbb47ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:07.835757 18180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bcf8b7bff14b6f943ddabd57016006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.838160 18391 tablet_bootstrap.cc:492] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: Bootstrap starting.
I20260925 13:16:07.838445 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.829331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39747 (local address 127.0.0.1:50052)
0925 13:16:07.830420 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:07.830430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:07.830478 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:07.830697 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:07.830698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:07.830701 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:07.830734 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.830735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.831965 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.831966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.833116 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:07.833118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.834170 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.834172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:07.834250 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.834254 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:07.838282 (+ 4028us) client_negotiation.cc:240] Negotiation successful
0925 13:16:07.838362 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260925 13:16:07.838812 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:07.829622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.0.98.3:39747)
0925 13:16:07.830354 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:07.830357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:07.830535 (+ 178us) server_negotiation.cc:407] Connection header received
0925 13:16:07.830594 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:07.830596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:07.830610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:07.830684 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:07.830816 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.831885 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.833206 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:07.833978 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:07.834048 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:07.834130 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:07.838659 (+ 4529us) server_negotiation.cc:299] Negotiation successful
0925 13:16:07.838732 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":138,"threads_started":1}
I20260925 13:16:07.838985 18392 tablet_bootstrap.cc:492] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Bootstrap starting.
I20260925 13:16:07.839262 18391 tablet_bootstrap.cc:654] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.839577 18391 log.cc:824] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.839735 18392 tablet_bootstrap.cc:654] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.839620 18311 tablet_service.cc:1467] Processing CreateTablet for tablet a9bcf8b7bff14b6f943ddabd57016006 (DEFAULT_TABLE table=test-table [id=987cbfa1ca3f4ac9894f86a7dbb47ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:07.840052 18311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bcf8b7bff14b6f943ddabd57016006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:07.840085 18392 log.cc:824] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.840986 18392 tablet_bootstrap.cc:492] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: No bootstrap required, opened a new log
I20260925 13:16:07.841075 18392 ts_tablet_manager.cc:1397] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:07.841141 18391 tablet_bootstrap.cc:492] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: No bootstrap required, opened a new log
I20260925 13:16:07.841223 18391 ts_tablet_manager.cc:1397] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 13:16:07.843017 18392 raft_consensus.cc:348] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:07.843230 18392 raft_consensus.cc:374] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.843276 18392 raft_consensus.cc:729] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75607f90588641c5b4e250282373dd6b, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.843408 18392 consensus_queue.cc:260] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:07.843722 18392 ts_tablet_manager.cc:1428] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:07.843866 18246 heartbeater.cc:503] Master 127.0.98.62:46019 was elected leader, sending a full tablet report...
I20260925 13:16:07.843957 18396 tablet_bootstrap.cc:492] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Bootstrap starting.
I20260925 13:16:07.844812 18396 tablet_bootstrap.cc:654] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:07.844754 18391 raft_consensus.cc:348] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:07.844975 18391 raft_consensus.cc:374] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.845023 18391 raft_consensus.cc:729] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0cbf2f4c5be4e53981f52556bb1a77b, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.845178 18396 log.cc:824] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:07.845151 18391 consensus_queue.cc:260] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:07.845494 18391 ts_tablet_manager.cc:1428] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 13:16:07.845546 18115 heartbeater.cc:503] Master 127.0.98.62:46019 was elected leader, sending a full tablet report...
I20260925 13:16:07.846131 18396 tablet_bootstrap.cc:492] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: No bootstrap required, opened a new log
I20260925 13:16:07.846212 18396 ts_tablet_manager.cc:1397] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:07.848305 18396 raft_consensus.cc:348] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:07.848491 18396 raft_consensus.cc:374] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:07.848529 18396 raft_consensus.cc:729] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bb9cabbe8344c69c864bbf6fbcf5a0, State: Initialized, Role: FOLLOWER
I20260925 13:16:07.848645 18396 consensus_queue.cc:260] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:07.848961 18396 ts_tablet_manager.cc:1428] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:07.849030 18377 heartbeater.cc:503] Master 127.0.98.62:46019 was elected leader, sending a full tablet report...
W20260925 13:16:07.923826 18247 tablet.cc:2369] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:16:08.034584 18116 tablet.cc:2369] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:16:08.058741 18378 tablet.cc:2369] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:08.076713 18399 raft_consensus.cc:482] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:16:08.076898 18399 raft_consensus.cc:504] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:08.077316 18399 leader_election.cc:288] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0cbf2f4c5be4e53981f52556bb1a77b (127.0.98.1:35077), 75607f90588641c5b4e250282373dd6b (127.0.98.2:39391)
I20260925 13:16:08.080289 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.077584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49127 (local address 127.0.98.1:35077)
0925 13:16:08.077634 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:08.077637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:08.077719 (+ 82us) server_negotiation.cc:407] Connection header received
0925 13:16:08.077811 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:08.077812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:08.077827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:08.077904 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:08.078069 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.078561 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.079566 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.079929 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.080017 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:08.080028 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:08.080175 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 13:16:08.080215 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:16:08.080289 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.077533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35077 (local address 127.0.98.3:49127)
0925 13:16:08.077621 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:08.077634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:08.077732 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:08.077909 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:08.077910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:08.077912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:08.077984 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:08.077986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.078652 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.078653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:08.079423 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:08.079426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.080025 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.080026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:08.080099 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:08.080102 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:08.080168 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 13:16:08.080215 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 13:16:08.080363 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.077631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43839 (local address 127.0.98.2:39391)
0925 13:16:08.077698 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:08.077701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:08.077823 (+ 122us) server_negotiation.cc:407] Connection header received
0925 13:16:08.077920 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:08.077921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:08.077930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:08.078014 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:08.078169 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.078632 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.079626 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.079992 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.080071 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:08.080079 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:08.080246 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 13:16:08.080285 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:16:08.080608 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.077547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39391 (local address 127.0.98.3:43839)
0925 13:16:08.077746 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:08.077755 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:08.077840 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:08.078020 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:08.078021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:08.078023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:08.078100 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:08.078102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.078718 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.078719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:08.079522 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:08.079524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.080073 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.080074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:08.080159 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:08.080161 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:16:08.080237 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:16:08.080278 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260925 13:16:08.080655 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bcf8b7bff14b6f943ddabd57016006" candidate_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" is_pre_election: true
I20260925 13:16:08.080859 18069 raft_consensus.cc:2454] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb9cabbe8344c69c864bbf6fbcf5a0 in term 0.
I20260925 13:16:08.080960 18200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bcf8b7bff14b6f943ddabd57016006" candidate_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75607f90588641c5b4e250282373dd6b" is_pre_election: true
I20260925 13:16:08.081132 18265 leader_election.cc:302] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0cbf2f4c5be4e53981f52556bb1a77b, b4bb9cabbe8344c69c864bbf6fbcf5a0; no voters:
I20260925 13:16:08.081152 18200 raft_consensus.cc:2454] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb9cabbe8344c69c864bbf6fbcf5a0 in term 0.
I20260925 13:16:08.081292 18399 raft_consensus.cc:2781] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:16:08.081357 18399 raft_consensus.cc:482] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:16:08.081382 18399 raft_consensus.cc:3037] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:08.082194 18399 raft_consensus.cc:504] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:08.082356 18399 leader_election.cc:288] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [CANDIDATE]: Term 1 election: Requested vote from peers a0cbf2f4c5be4e53981f52556bb1a77b (127.0.98.1:35077), 75607f90588641c5b4e250282373dd6b (127.0.98.2:39391)
I20260925 13:16:08.082515 18200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bcf8b7bff14b6f943ddabd57016006" candidate_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75607f90588641c5b4e250282373dd6b"
I20260925 13:16:08.082538 18069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bcf8b7bff14b6f943ddabd57016006" candidate_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b"
I20260925 13:16:08.082605 18200 raft_consensus.cc:3037] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:08.082613 18069 raft_consensus.cc:3037] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:08.083350 18069 raft_consensus.cc:2454] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb9cabbe8344c69c864bbf6fbcf5a0 in term 1.
I20260925 13:16:08.083437 18200 raft_consensus.cc:2454] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb9cabbe8344c69c864bbf6fbcf5a0 in term 1.
I20260925 13:16:08.083526 18265 leader_election.cc:302] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0cbf2f4c5be4e53981f52556bb1a77b, b4bb9cabbe8344c69c864bbf6fbcf5a0; no voters:
I20260925 13:16:08.083629 18399 raft_consensus.cc:2781] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:08.083776 18399 raft_consensus.cc:686] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 1 LEADER]: Becoming Leader. State: Replica: b4bb9cabbe8344c69c864bbf6fbcf5a0, State: Running, Role: LEADER
I20260925 13:16:08.083887 18399 consensus_queue.cc:237] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } }
I20260925 13:16:08.084756 17926 catalog_manager.cc:5495] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4bb9cabbe8344c69c864bbf6fbcf5a0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb9cabbe8344c69c864bbf6fbcf5a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 } health_report { overall_health: UNKNOWN } } }
I20260925 13:16:08.120550 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.118302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.98.3:39747)
0925 13:16:08.118344 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:08.118345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:08.118352 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:16:08.118417 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:08.118418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:08.118443 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:08.118517 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:08.118635 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.119091 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.119840 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:08.120034 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:08.120089 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:08.120213 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:08.120313 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:08.120370 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:08.120407 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:16:08.120421 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:16:08.122645 18069 raft_consensus.cc:1261] T a9bcf8b7bff14b6f943ddabd57016006 P a0cbf2f4c5be4e53981f52556bb1a77b [term 1 FOLLOWER]: Refusing update from remote peer b4bb9cabbe8344c69c864bbf6fbcf5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:08.122645 18200 raft_consensus.cc:1261] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 1 FOLLOWER]: Refusing update from remote peer b4bb9cabbe8344c69c864bbf6fbcf5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:08.123174 18399 consensus_queue.cc:1035] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75607f90588641c5b4e250282373dd6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:08.123350 18401 consensus_queue.cc:1035] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0cbf2f4c5be4e53981f52556bb1a77b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:08.126057 18405 mvcc.cc:205] Tried to move back new op lower bound from 7333241520627425280 to 7333241520472408064. Current Snapshot: MvccSnapshot[applied={T|T < 7333241520627425280}]
I20260925 13:16:08.127341 18406 mvcc.cc:205] Tried to move back new op lower bound from 7333241520627425280 to 7333241520472408064. Current Snapshot: MvccSnapshot[applied={T|T < 7333241520627425280 or (T in {7333241520627425280})}]
I20260925 13:16:08.128614 17927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:08.128721 17927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:08.129504 17927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d7dc206139b4ac7bdea8e49d83288f2: Sending DeleteTablet for 3 replicas of tablet a9bcf8b7bff14b6f943ddabd57016006
I20260925 13:16:08.129837 18311 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bcf8b7bff14b6f943ddabd57016006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:08 UTC) from {username='slave'} at 127.0.0.1:50052
I20260925 13:16:08.129876 18180 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bcf8b7bff14b6f943ddabd57016006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:08 UTC) from {username='slave'} at 127.0.0.1:56934
I20260925 13:16:08.130220 18414 tablet_replica.cc:330] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: stopping tablet replica
I20260925 13:16:08.130228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17987
I20260925 13:16:08.130214 18049 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bcf8b7bff14b6f943ddabd57016006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:08 UTC) from {username='slave'} at 127.0.0.1:54578
I20260925 13:16:08.130357 18414 raft_consensus.cc:2229] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:16:08.130842 18414 raft_consensus.cc:2258] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:08.131736 18414 ts_tablet_manager.cc:1905] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:08.133795 18414 ts_tablet_manager.cc:1918] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:08.133805 18415 tablet_replica.cc:330] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: stopping tablet replica
I20260925 13:16:08.133939 18415 raft_consensus.cc:2229] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:08.134039 18414 log.cc:1196] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/a9bcf8b7bff14b6f943ddabd57016006
I20260925 13:16:08.134073 18415 raft_consensus.cc:2258] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:08.134398 18414 ts_tablet_manager.cc:1939] T a9bcf8b7bff14b6f943ddabd57016006 P b4bb9cabbe8344c69c864bbf6fbcf5a0: Deleting consensus metadata
I20260925 13:16:08.134545 18415 ts_tablet_manager.cc:1905] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:08.134933 17913 catalog_manager.cc:4841] TS b4bb9cabbe8344c69c864bbf6fbcf5a0 (127.0.98.3:39747): tablet a9bcf8b7bff14b6f943ddabd57016006 (table test-table [id=987cbfa1ca3f4ac9894f86a7dbb47ee3]) successfully deleted
I20260925 13:16:08.136417 18415 ts_tablet_manager.cc:1918] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:08.136497 18415 log.cc:1196] T a9bcf8b7bff14b6f943ddabd57016006 P 75607f90588641c5b4e250282373dd6b: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/a9bcf8b7bff14b6f943ddabd57016006
W20260925 13:16:08.136552 17912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35077 (error 108)
I20260925 13:16:08.136690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18118
I20260925 13:16:08.138721 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.138540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35077 (local address 127.0.0.1:54594)
0925 13:16:08.138619 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:08.138681 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 13:16:08.138919 17912 catalog_manager.cc:4584] TS a0cbf2f4c5be4e53981f52556bb1a77b (127.0.98.1:35077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9bcf8b7bff14b6f943ddabd57016006: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35077: connect: Connection refused (error 111)
I20260925 13:16:08.141469 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:08.141329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39391 (local address 127.0.0.1:56950)
0925 13:16:08.141397 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:08.141429 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:16:08.142091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18249
I20260925 13:16:08.146838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 17896
2026-09-25T13:16:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:08 dist-test-slave-2d0s krb5kdc[18419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:08 dist-test-slave-2d0s krb5kdc[18419](info): set up 2 sockets
Sep 25 13:16:08 dist-test-slave-2d0s krb5kdc[18419](info): commencing operation
krb5kdc: starting...
W20260925 13:16:09.189450 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:09.312640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46351
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:09.396893 18432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:09.397297 18432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:09.397374 18432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:09.398758 18432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:09.398844 18432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:09.398867 18432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:09.398887 18432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:09.398907 18432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:09.400625 18432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:09.400971 18432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:09.401229 18432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:09.406992 18440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:09.406992 18438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:09.407028 18432 server_base.cc:1034] running on GCE node
W20260925 13:16:09.407008 18437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:09.407490 18432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:09.407820 18432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:09.408981 18432 hybrid_clock.cc:648] HybridClock initialized: now 1790342169408954 us; error 39 us; skew 500 ppm
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:09.418519 18432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:09.419113 18432 webserver.cc:460] Webserver started at http://127.0.98.62:34847/ using document root <none> and password file <none>
I20260925 13:16:09.419359 18432 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:09.419438 18432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:09.419597 18432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:09.420742 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "8268ffca655d4f218e28aed5511024bb"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "1f0f17417689b7ee1afe3c742e612fc1"
server_key_iv: "63b6d83c44cae87c132aa766b23f43b0"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.421123 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "8268ffca655d4f218e28aed5511024bb"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "1f0f17417689b7ee1afe3c742e612fc1"
server_key_iv: "63b6d83c44cae87c132aa766b23f43b0"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.422658 18432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:09.423456 18447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.423650 18432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:09.423763 18432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "8268ffca655d4f218e28aed5511024bb"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "1f0f17417689b7ee1afe3c742e612fc1"
server_key_iv: "63b6d83c44cae87c132aa766b23f43b0"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.423844 18432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:09.445942 18432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:09.448112 18432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:09.448312 18432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:09.452723 18432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46351
I20260925 13:16:09.452765 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46351 every 8 connection(s)
I20260925 13:16:09.453207 18432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:09.454193 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:09.456723 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb: Bootstrap starting.
I20260925 13:16:09.457386 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:09.457724 18500 log.cc:824] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:09.458503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18432
I20260925 13:16:09.458614 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb: No bootstrap required, opened a new log
I20260925 13:16:09.458630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:09.458864 392 external_mini_cluster.cc:1428] Setting key 35253d6b5ca39dc430d4165e044b05eb
I20260925 13:16:09.460647 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8268ffca655d4f218e28aed5511024bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46351 } }
I20260925 13:16:09.460834 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:09.460872 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8268ffca655d4f218e28aed5511024bb, State: Initialized, Role: FOLLOWER
I20260925 13:16:09.461014 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8268ffca655d4f218e28aed5511024bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46351 } }
I20260925 13:16:09.461109 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:09.461145 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:09.461197 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:09.462402 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8268ffca655d4f218e28aed5511024bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46351 } }
I20260925 13:16:09.462503 18500 leader_election.cc:302] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8268ffca655d4f218e28aed5511024bb; no voters:
I20260925 13:16:09.462699 18500 leader_election.cc:288] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:09.462754 18505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:09.462966 18505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [term 1 LEADER]: Becoming Leader. State: Replica: 8268ffca655d4f218e28aed5511024bb, State: Running, Role: LEADER
I20260925 13:16:09.463129 18505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8268ffca655d4f218e28aed5511024bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46351 } }
I20260925 13:16:09.463578 18507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8268ffca655d4f218e28aed5511024bb. Latest consensus state: current_term: 1 leader_uuid: "8268ffca655d4f218e28aed5511024bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8268ffca655d4f218e28aed5511024bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46351 } } }
I20260925 13:16:09.463618 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8268ffca655d4f218e28aed5511024bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8268ffca655d4f218e28aed5511024bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46351 } } }
I20260925 13:16:09.463745 18507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:09.463763 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:09.463967 18509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:09.464533 18509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:09.462996 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:09.467577 18509 catalog_manager.cc:1347] Generated new cluster ID: 6583e60ecf5e464ba4fef4219e303c85
I20260925 13:16:09.467633 18509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:09.472458 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.459513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.98.62:46351)
0925 13:16:09.459721 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:09.459737 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:09.459765 (+ 28us) server_negotiation.cc:407] Connection header received
0925 13:16:09.460633 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:09.460642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:09.460703 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:09.460879 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:09.461772 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.463003 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.464117 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.464738 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.464851 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:09.467893 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:09.467898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:09.467899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:09.467912 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:09.470012 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:09.470468 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:09.470472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:09.470473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:09.470613 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:09.470841 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:09.470842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:09.470843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:09.471194 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:09.471361 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:09.471860 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 13:16:09.472015 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260925 13:16:09.489946 18509 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:09.490585 18509 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:09.493026 18509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb: Generated new TSK 0
I20260925 13:16:09.493245 18509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:16:09.528604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46351
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:09.630077 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:09.630329 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:09.630357 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:09.631578 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:09.631624 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:09.631652 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:16:09.633178 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:09.633414 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:09.633608 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:09.639025 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:09.639190 18526 server_base.cc:1034] running on GCE node
W20260925 13:16:09.639007 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:09.639014 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:09.639585 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:09.639832 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:09.641047 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790342169641024 us; error 38 us; skew 500 ppm
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:09.643791 18526 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:16:09.644217 18526 webserver.cc:460] Webserver started at http://127.0.98.1:45781/ using document root <none> and password file <none>
I20260925 13:16:09.644426 18526 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:09.644469 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:09.644562 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:09.645615 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea64c33ab0d54b28bd0af347008a7e8d"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "99053bb00e3c5c32a8a416720226345e"
server_key_iv: "ea53113ff4d873e1fbc7c3b2a205924f"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.646027 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea64c33ab0d54b28bd0af347008a7e8d"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "99053bb00e3c5c32a8a416720226345e"
server_key_iv: "ea53113ff4d873e1fbc7c3b2a205924f"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.647437 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:09.648232 18541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.648430 18526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:09.648545 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "ea64c33ab0d54b28bd0af347008a7e8d"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "99053bb00e3c5c32a8a416720226345e"
server_key_iv: "ea53113ff4d873e1fbc7c3b2a205924f"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.648626 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:09.672107 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:09.674374 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:09.674566 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:09.674836 18526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:09.675195 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:09.675252 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.675295 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:09.675325 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.681388 18526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44747
I20260925 13:16:09.681437 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44747 every 8 connection(s)
I20260925 13:16:09.681806 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:16:09.684899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18526
I20260925 13:16:09.685042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:16:09.685324 392 external_mini_cluster.cc:1428] Setting key b32f119a24167618828e3c58280c1e74
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:09.693303 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.682574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43763 (local address 127.0.98.62:46351)
0925 13:16:09.682639 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:09.682641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:09.682882 (+ 241us) server_negotiation.cc:407] Connection header received
0925 13:16:09.683862 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:09.683864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:09.683875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:09.683937 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:09.684568 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.685130 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.686658 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.686872 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.686931 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:09.689564 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:09.689566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:09.689566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:09.689576 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:09.691533 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:09.692158 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:09.692159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:09.692160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:09.692247 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:09.692566 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:09.692567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:09.692568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:09.692722 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:09.692802 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:09.693066 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 13:16:09.693178 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:16:09.693336 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.682535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46351 (local address 127.0.98.1:43763)
0925 13:16:09.682746 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:09.682791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:09.683753 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:09.684004 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:09.684008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:09.684282 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:09.684487 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:09.684491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.685251 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.685252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:09.686571 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:09.686574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.686958 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.686959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:09.687109 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:09.687768 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:09.687777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:09.689450 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:09.691655 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:09.691664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:09.691667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:09.692068 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:09.692337 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:09.692338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:09.692339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:09.692483 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:09.692795 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:09.692800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:09.692931 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 13:16:09.693019 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 13:16:09.693152 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260925 13:16:09.694253 18655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46351
I20260925 13:16:09.694358 18655 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:09.694514 18655 heartbeater.cc:511] Master 127.0.98.62:46351 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:16:09.695068 18464 ts_manager.cc:194] Registered new tserver with Master: ea64c33ab0d54b28bd0af347008a7e8d (127.0.98.1:44747)
I20260925 13:16:09.695698 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:16:09.713443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46351
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:09.795673 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:09.795950 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:09.795993 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:09.797223 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:09.797302 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:09.797349 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:16:09.798957 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:09.799256 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:09.799504 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:09.804611 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:09.804677 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:09.804625 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:09.804996 18660 server_base.cc:1034] running on GCE node
I20260925 13:16:09.805227 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:09.805488 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:09.806631 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790342169806591 us; error 59 us; skew 500 ppm
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:09.809087 18660 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:16:09.809537 18660 webserver.cc:460] Webserver started at http://127.0.98.2:35219/ using document root <none> and password file <none>
I20260925 13:16:09.809813 18660 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:09.809885 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:09.810045 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:09.811141 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "55e2edb45b3944bebe76997a2dce621f"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "fc653b607cee6d980f7f35221768a8b7"
server_key_iv: "e33f01616f45562201ddf1fc3f0b1d0a"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.811523 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "55e2edb45b3944bebe76997a2dce621f"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "fc653b607cee6d980f7f35221768a8b7"
server_key_iv: "e33f01616f45562201ddf1fc3f0b1d0a"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.812911 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:09.813710 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.813894 18660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:09.813995 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "55e2edb45b3944bebe76997a2dce621f"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "fc653b607cee6d980f7f35221768a8b7"
server_key_iv: "e33f01616f45562201ddf1fc3f0b1d0a"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.814085 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:09.829404 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:09.831498 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:09.831615 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:09.831805 18660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:09.832127 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:09.832165 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.832188 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:09.832204 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.838259 18660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41345
I20260925 13:16:09.838354 18788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41345 every 8 connection(s)
I20260925 13:16:09.838677 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:09.847893 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.839489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48685 (local address 127.0.98.62:46351)
0925 13:16:09.839559 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:09.839561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:09.839749 (+ 188us) server_negotiation.cc:407] Connection header received
0925 13:16:09.840695 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:09.840696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:09.840708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:09.840777 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:09.841376 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.841979 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.842987 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.843190 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.843255 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:09.845362 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:09.845364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:09.845364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:09.845371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:09.846698 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:09.847069 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:09.847070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:09.847070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:09.847127 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:09.847364 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:09.847364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:09.847365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:09.847472 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:09.847523 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:09.847753 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 13:16:09.847822 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:16:09.848075 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.839452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46351 (local address 127.0.98.2:48685)
0925 13:16:09.839609 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:09.839651 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:09.840585 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:09.840832 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:09.840836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:09.841086 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:09.841283 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:09.841287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.842066 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.842067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:09.842888 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:09.842890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.843271 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.843272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:09.843370 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:09.843866 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:09.843874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:09.845279 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:09.846782 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:09.846788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:09.846790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:09.847001 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:09.847195 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:09.847196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:09.847196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:09.847294 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:09.847527 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:09.847531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:09.847645 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 13:16:09.847727 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:16:09.847860 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260925 13:16:09.848634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18660
I20260925 13:16:09.848630 18789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46351
I20260925 13:16:09.848745 18789 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:09.848755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:16:09.848955 18789 heartbeater.cc:511] Master 127.0.98.62:46351 requested a full tablet report, sending...
I20260925 13:16:09.848986 392 external_mini_cluster.cc:1428] Setting key d64f114a56c447b225551f083d42829d
I20260925 13:16:09.849408 18463 ts_manager.cc:194] Registered new tserver with Master: 55e2edb45b3944bebe76997a2dce621f (127.0.98.2:41345)
I20260925 13:16:09.849905 18463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48685
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:16:09.874248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46351
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:09.957209 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:09.957480 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:09.957518 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:09.958761 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:09.958838 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:09.958881 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:16:09.960423 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:09.960727 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:09.960968 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:09.965973 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:09.966038 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:09.965989 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:09.966439 18794 server_base.cc:1034] running on GCE node
I20260925 13:16:09.966665 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:09.966928 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:09.968089 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790342169968080 us; error 31 us; skew 500 ppm
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:09.971000 18794 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:16:09.971567 18794 webserver.cc:460] Webserver started at http://127.0.98.3:42781/ using document root <none> and password file <none>
I20260925 13:16:09.971863 18794 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:09.971936 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:09.972087 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:09.973222 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "99eb66e53cf4450bba913db6388c60dc"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "8f40d802cf445d477551f072ea907734"
server_key_iv: "92c96b9148648a2179131868a75e20e1"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.973609 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "99eb66e53cf4450bba913db6388c60dc"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "8f40d802cf445d477551f072ea907734"
server_key_iv: "92c96b9148648a2179131868a75e20e1"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.975018 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:09.975894 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.976123 18794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:16:09.976233 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "99eb66e53cf4450bba913db6388c60dc"
format_stamp: "Formatted at 2026-09-25 13:16:09 on dist-test-slave-2d0s"
server_key: "8f40d802cf445d477551f072ea907734"
server_key_iv: "92c96b9148648a2179131868a75e20e1"
server_key_version: "encryptionkey@0"
I20260925 13:16:09.976316 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:09.986063 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:09.988521 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:09.988694 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:09.988940 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:09.989290 18794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:09.989346 18794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.989392 18794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:09.989419 18794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:09.995325 18794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45445
I20260925 13:16:09.995393 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45445 every 8 connection(s)
I20260925 13:16:09.995752 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:16:09.999384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18794
I20260925 13:16:09.999514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:16:09.999738 392 external_mini_cluster.cc:1428] Setting key a56af228e56e776d5f7bda58c0ba5d1e
Sep 25 13:16:09 dist-test-slave-2d0s krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:10.006026 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.996556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35815 (local address 127.0.98.62:46351)
0925 13:16:09.996608 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:09.996610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:09.996846 (+ 236us) server_negotiation.cc:407] Connection header received
0925 13:16:09.997918 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:09.997920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:09.997931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:09.997997 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:09.998688 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.999185 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.000542 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.000838 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.000929 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.003234 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.003236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.003236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:10.003244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.004610 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:10.005040 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:10.005041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:10.005042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:10.005115 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:10.005392 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:10.005393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:10.005394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:10.005510 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.005567 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.005880 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.005953 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:16:10.006217 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:09.996516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46351 (local address 127.0.98.3:35815)
0925 13:16:09.996703 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:09.996747 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:09.997812 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:09.998068 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:09.998072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:09.998365 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:09.998575 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:09.998580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:09.999289 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:09.999290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.000463 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.000466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.000937 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.000938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.001059 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.001576 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:10.001585 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:10.003133 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:10.004707 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:10.004714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:10.004716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:10.004935 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:10.005200 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:10.005201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:10.005201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:10.005330 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:10.005569 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:10.005573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:10.005759 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.005850 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.005977 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260925 13:16:10.006784 18923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46351
I20260925 13:16:10.006893 18923 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:10.007073 18923 heartbeater.cc:511] Master 127.0.98.62:46351 requested a full tablet report, sending...
I20260925 13:16:10.007479 18463 ts_manager.cc:194] Registered new tserver with Master: 99eb66e53cf4450bba913db6388c60dc (127.0.98.3:45445)
I20260925 13:16:10.007884 18463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35815
I20260925 13:16:10.011617 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:16:10.020934 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.012921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.98.62:46351)
0925 13:16:10.012988 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.012990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.013112 (+ 122us) server_negotiation.cc:407] Connection header received
0925 13:16:10.013191 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.013193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.013203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.013274 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:10.013914 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.014407 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.015154 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.015356 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.015431 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.016968 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.016969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.016970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:10.016978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.019140 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:10.019762 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:10.019763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:10.019764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:10.019847 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:10.020157 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:10.020158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:10.020159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:10.020382 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.020503 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.020722 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.020800 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:16:10.024070 18463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:16:10.026415 18463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:16:10.033780 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.029466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.98.1:44747)
0925 13:16:10.029623 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.029629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.029868 (+ 239us) server_negotiation.cc:407] Connection header received
0925 13:16:10.029981 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.029984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.030002 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.030094 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.030320 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.031512 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.032804 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.033342 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.033430 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.033495 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.033626 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.033692 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260925 13:16:10.033774 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.029228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44747 (local address 127.0.0.1:48614)
0925 13:16:10.029752 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.029783 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.029877 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.030104 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.030107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.030110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.030223 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.030226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.031652 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.031653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.032698 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.032701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.033430 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.033431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.033519 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.033533 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.033606 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.033688 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":41,"thread_start_us":74,"threads_started":1}
I20260925 13:16:10.033939 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.029451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41345 (local address 127.0.0.1:39322)
0925 13:16:10.029986 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.029996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.030038 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.030339 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.030339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.030341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.030417 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.030419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.031325 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.031326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.032567 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.032570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.033634 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.033635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.033724 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.033729 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.033785 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.033836 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260925 13:16:10.034235 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.029599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.98.2:41345)
0925 13:16:10.029847 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.029850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.030100 (+ 250us) server_negotiation.cc:407] Connection header received
0925 13:16:10.030151 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.030154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.030169 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.030238 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.030502 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.031195 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.032674 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.033438 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.033521 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.033581 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.034097 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.034151 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":125,"threads_started":1}
I20260925 13:16:10.034469 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.029228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45445 (local address 127.0.0.1:50260)
0925 13:16:10.029731 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.029778 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.029872 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.030334 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.030335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.030337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.030417 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.030419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.031681 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.031682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.032817 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.032819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.033709 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.033710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.033769 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.033773 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.033818 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.034400 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260925 13:16:10.034791 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.029356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.0.98.3:45445)
0925 13:16:10.029724 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.029730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.029899 (+ 169us) server_negotiation.cc:407] Connection header received
0925 13:16:10.029978 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.029981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.030001 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.030241 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.030485 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.031567 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.032883 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.033573 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.033642 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.034137 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.034165 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.034209 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260925 13:16:10.034780 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 0fefd769cbac4dd5b8b51cbd0566d268 (DEFAULT_TABLE table=test-table [id=1718e54cdc7b4f378ee3ae5e4d13ae2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:10.034905 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 0fefd769cbac4dd5b8b51cbd0566d268 (DEFAULT_TABLE table=test-table [id=1718e54cdc7b4f378ee3ae5e4d13ae2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:10.035157 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fefd769cbac4dd5b8b51cbd0566d268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.035190 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fefd769cbac4dd5b8b51cbd0566d268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.035589 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 0fefd769cbac4dd5b8b51cbd0566d268 (DEFAULT_TABLE table=test-table [id=1718e54cdc7b4f378ee3ae5e4d13ae2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:10.035842 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fefd769cbac4dd5b8b51cbd0566d268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.038583 18940 tablet_bootstrap.cc:492] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Bootstrap starting.
I20260925 13:16:10.039021 18942 tablet_bootstrap.cc:492] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Bootstrap starting.
I20260925 13:16:10.039309 18940 tablet_bootstrap.cc:654] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.039584 18942 tablet_bootstrap.cc:654] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.039871 18940 log.cc:824] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.040066 18942 log.cc:824] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.040256 18941 tablet_bootstrap.cc:492] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: Bootstrap starting.
I20260925 13:16:10.040709 18940 tablet_bootstrap.cc:492] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: No bootstrap required, opened a new log
I20260925 13:16:10.040795 18940 ts_tablet_manager.cc:1397] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:10.040841 18942 tablet_bootstrap.cc:492] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: No bootstrap required, opened a new log
I20260925 13:16:10.040908 18942 ts_tablet_manager.cc:1397] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:10.040928 18941 tablet_bootstrap.cc:654] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.041602 18941 log.cc:824] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.042351 18941 tablet_bootstrap.cc:492] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: No bootstrap required, opened a new log
I20260925 13:16:10.042438 18941 ts_tablet_manager.cc:1397] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:10.042855 18940 raft_consensus.cc:348] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.042860 18942 raft_consensus.cc:348] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.043046 18940 raft_consensus.cc:374] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.043046 18942 raft_consensus.cc:374] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.043084 18940 raft_consensus.cc:729] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99eb66e53cf4450bba913db6388c60dc, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.043084 18942 raft_consensus.cc:729] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e2edb45b3944bebe76997a2dce621f, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.043210 18942 consensus_queue.cc:260] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.043247 18940 consensus_queue.cc:260] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.043516 18942 ts_tablet_manager.cc:1428] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:10.043516 18940 ts_tablet_manager.cc:1428] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:10.043632 18923 heartbeater.cc:503] Master 127.0.98.62:46351 was elected leader, sending a full tablet report...
I20260925 13:16:10.043699 18789 heartbeater.cc:503] Master 127.0.98.62:46351 was elected leader, sending a full tablet report...
I20260925 13:16:10.044488 18941 raft_consensus.cc:348] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.044691 18941 raft_consensus.cc:374] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.044785 18941 raft_consensus.cc:729] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea64c33ab0d54b28bd0af347008a7e8d, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.044914 18941 consensus_queue.cc:260] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.045228 18941 ts_tablet_manager.cc:1428] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:10.045264 18655 heartbeater.cc:503] Master 127.0.98.62:46351 was elected leader, sending a full tablet report...
W20260925 13:16:10.089250 18790 tablet.cc:2369] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:10.156785 18948 raft_consensus.cc:482] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:16:10.156940 18948 raft_consensus.cc:504] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.157287 18948 leader_election.cc:288] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99eb66e53cf4450bba913db6388c60dc (127.0.98.3:45445), 55e2edb45b3944bebe76997a2dce621f (127.0.98.2:41345)
I20260925 13:16:10.160355 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.157588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44983 (local address 127.0.98.3:45445)
0925 13:16:10.157638 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.157640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.157725 (+ 85us) server_negotiation.cc:407] Connection header received
0925 13:16:10.157788 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.157789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.157801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.157881 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.158074 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.158617 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.159604 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.159988 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.160043 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.160077 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.160236 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.160283 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:16:10.160354 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.157523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45445 (local address 127.0.98.1:44983)
0925 13:16:10.157600 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.157640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.157718 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.157884 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.157885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.157887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.157965 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.157967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.158701 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.158702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.159510 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.159513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.160076 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.160077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.160164 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.160167 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.160209 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.160265 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 13:16:10.160403 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.157583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33137 (local address 127.0.98.2:41345)
0925 13:16:10.157692 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.157694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.157920 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:16:10.157951 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.157952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.157964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.158021 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.158237 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.158720 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.159641 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.159997 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.160089 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.160098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.160279 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.160326 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 13:16:10.160372 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.157523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41345 (local address 127.0.98.1:33137)
0925 13:16:10.157696 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.157706 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.157784 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.158074 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.158075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.158078 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.158151 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.158153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.158803 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.158803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.159527 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.159528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.160117 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.160118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.160182 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.160186 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.160259 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.160293 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":15,"thread_start_us":61,"threads_started":1}
I20260925 13:16:10.160784 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fefd769cbac4dd5b8b51cbd0566d268" candidate_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99eb66e53cf4450bba913db6388c60dc" is_pre_election: true
I20260925 13:16:10.160991 18877 raft_consensus.cc:2454] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea64c33ab0d54b28bd0af347008a7e8d in term 0.
I20260925 13:16:10.161029 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fefd769cbac4dd5b8b51cbd0566d268" candidate_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e2edb45b3944bebe76997a2dce621f" is_pre_election: true
I20260925 13:16:10.161216 18743 raft_consensus.cc:2454] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea64c33ab0d54b28bd0af347008a7e8d in term 0.
I20260925 13:16:10.161266 18545 leader_election.cc:302] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99eb66e53cf4450bba913db6388c60dc, ea64c33ab0d54b28bd0af347008a7e8d; no voters:
I20260925 13:16:10.161423 18948 raft_consensus.cc:2781] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:16:10.161502 18948 raft_consensus.cc:482] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:16:10.161540 18948 raft_consensus.cc:3037] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.162323 18948 raft_consensus.cc:504] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.162493 18948 leader_election.cc:288] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [CANDIDATE]: Term 1 election: Requested vote from peers 99eb66e53cf4450bba913db6388c60dc (127.0.98.3:45445), 55e2edb45b3944bebe76997a2dce621f (127.0.98.2:41345)
I20260925 13:16:10.162663 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fefd769cbac4dd5b8b51cbd0566d268" candidate_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99eb66e53cf4450bba913db6388c60dc"
I20260925 13:16:10.162696 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fefd769cbac4dd5b8b51cbd0566d268" candidate_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e2edb45b3944bebe76997a2dce621f"
I20260925 13:16:10.162736 18877 raft_consensus.cc:3037] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.162766 18743 raft_consensus.cc:3037] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.163403 18743 raft_consensus.cc:2454] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea64c33ab0d54b28bd0af347008a7e8d in term 1.
I20260925 13:16:10.163407 18877 raft_consensus.cc:2454] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea64c33ab0d54b28bd0af347008a7e8d in term 1.
I20260925 13:16:10.163589 18545 leader_election.cc:302] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99eb66e53cf4450bba913db6388c60dc, ea64c33ab0d54b28bd0af347008a7e8d; no voters:
I20260925 13:16:10.163704 18948 raft_consensus.cc:2781] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:10.163851 18948 raft_consensus.cc:686] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [term 1 LEADER]: Becoming Leader. State: Replica: ea64c33ab0d54b28bd0af347008a7e8d, State: Running, Role: LEADER
I20260925 13:16:10.163946 18948 consensus_queue.cc:237] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260925 13:16:10.164870 18463 catalog_manager.cc:5495] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d reported cstate change: term changed from 0 to 1, leader changed from <none> to ea64c33ab0d54b28bd0af347008a7e8d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea64c33ab0d54b28bd0af347008a7e8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } health_report { overall_health: UNKNOWN } } }
I20260925 13:16:10.179338 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.176952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.0.98.1:44747)
0925 13:16:10.177015 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.177017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.177047 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:16:10.177084 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.177084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.177106 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.177161 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:10.177298 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.177847 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.178575 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.178768 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.178823 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.178949 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:10.179039 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:10.179158 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.179199 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.179212 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:16:10.181329 18877 raft_consensus.cc:1261] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 1 FOLLOWER]: Refusing update from remote peer ea64c33ab0d54b28bd0af347008a7e8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:10.181345 18743 raft_consensus.cc:1261] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 1 FOLLOWER]: Refusing update from remote peer ea64c33ab0d54b28bd0af347008a7e8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:10.181699 18948 consensus_queue.cc:1035] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e2edb45b3944bebe76997a2dce621f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:10.181849 18950 consensus_queue.cc:1035] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99eb66e53cf4450bba913db6388c60dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:16:10.183322 18656 tablet.cc:2369] T 0fefd769cbac4dd5b8b51cbd0566d268 P ea64c33ab0d54b28bd0af347008a7e8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:10.184598 18954 mvcc.cc:205] Tried to move back new op lower bound from 7333241529059713024 to 7333241528992423936. Current Snapshot: MvccSnapshot[applied={T|T < 7333241529059713024}]
I20260925 13:16:10.185851 18955 mvcc.cc:205] Tried to move back new op lower bound from 7333241529059713024 to 7333241528992423936. Current Snapshot: MvccSnapshot[applied={T|T < 7333241529059713024 or (T in {7333241529059713024})}]
I20260925 13:16:10.186625 18463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:10.186764 18463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:10.187537 18463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8268ffca655d4f218e28aed5511024bb: Sending DeleteTablet for 3 replicas of tablet 0fefd769cbac4dd5b8b51cbd0566d268
I20260925 13:16:10.187842 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fefd769cbac4dd5b8b51cbd0566d268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50260
I20260925 13:16:10.188006 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fefd769cbac4dd5b8b51cbd0566d268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48614
I20260925 13:16:10.188052 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fefd769cbac4dd5b8b51cbd0566d268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39322
I20260925 13:16:10.188277 18963 tablet_replica.cc:330] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: stopping tablet replica
I20260925 13:16:10.188271 18965 tablet_replica.cc:330] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: stopping tablet replica
I20260925 13:16:10.188381 18963 raft_consensus.cc:2229] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:10.188378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18526
I20260925 13:16:10.188499 18965 raft_consensus.cc:2229] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:10.188647 18965 raft_consensus.cc:2258] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:10.188982 18963 raft_consensus.cc:2258] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:10.189376 18963 ts_tablet_manager.cc:1905] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:10.190845 18965 ts_tablet_manager.cc:1905] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:10.191231 18963 ts_tablet_manager.cc:1918] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:10.191291 18963 log.cc:1196] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/0fefd769cbac4dd5b8b51cbd0566d268
I20260925 13:16:10.191527 18963 ts_tablet_manager.cc:1939] T 0fefd769cbac4dd5b8b51cbd0566d268 P 99eb66e53cf4450bba913db6388c60dc: Deleting consensus metadata
I20260925 13:16:10.191990 18451 catalog_manager.cc:4841] TS 99eb66e53cf4450bba913db6388c60dc (127.0.98.3:45445): tablet 0fefd769cbac4dd5b8b51cbd0566d268 (table test-table [id=1718e54cdc7b4f378ee3ae5e4d13ae2c]) successfully deleted
I20260925 13:16:10.192457 18965 ts_tablet_manager.cc:1918] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:10.192546 18965 log.cc:1196] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/0fefd769cbac4dd5b8b51cbd0566d268
I20260925 13:16:10.192790 18965 ts_tablet_manager.cc:1939] T 0fefd769cbac4dd5b8b51cbd0566d268 P 55e2edb45b3944bebe76997a2dce621f: Deleting consensus metadata
I20260925 13:16:10.193176 18450 catalog_manager.cc:4841] TS 55e2edb45b3944bebe76997a2dce621f (127.0.98.2:41345): tablet 0fefd769cbac4dd5b8b51cbd0566d268 (table test-table [id=1718e54cdc7b4f378ee3ae5e4d13ae2c]) successfully deleted
W20260925 13:16:10.196425 18450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44747 (error 108)
I20260925 13:16:10.196527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18660
I20260925 13:16:10.197019 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.196899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44747 (local address 127.0.0.1:48634)
0925 13:16:10.196951 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.196978 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:16:10.197135 18450 catalog_manager.cc:4584] TS ea64c33ab0d54b28bd0af347008a7e8d (127.0.98.1:44747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fefd769cbac4dd5b8b51cbd0566d268: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44747: connect: Connection refused (error 111)
I20260925 13:16:10.201481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18794
I20260925 13:16:10.206530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18432
2026-09-25T13:16:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2059 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T13:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:10Z Disabled control of system clock
I20260925 13:16:10.234395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44253
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:10.317226 18970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:10.317452 18970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:10.317474 18970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:10.318598 18970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:10.318634 18970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:10.318647 18970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:10.318660 18970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:10.318671 18970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:10.319948 18970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:10.320155 18970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:10.320401 18970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:10.325234 18975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:10.325254 18976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.325280 18970 server_base.cc:1034] running on GCE node
W20260925 13:16:10.325233 18978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.325727 18970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:10.326052 18970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:10.327221 18970 hybrid_clock.cc:648] HybridClock initialized: now 1790342170327205 us; error 27 us; skew 500 ppm
I20260925 13:16:10.328648 18970 webserver.cc:460] Webserver started at http://127.0.98.62:40977/ using document root <none> and password file <none>
I20260925 13:16:10.328913 18970 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:10.328989 18970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:10.329144 18970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:10.330366 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "85b1f472cf6542e5b293da4b59da172c"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "067eeffb9db11508f44c2b677f46aa8c"
server_key_iv: "ba273b6b17b0995a1d312ac2bbfb9722"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.330750 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "85b1f472cf6542e5b293da4b59da172c"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "067eeffb9db11508f44c2b677f46aa8c"
server_key_iv: "ba273b6b17b0995a1d312ac2bbfb9722"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.332083 18970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:10.332861 18984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.333063 18970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:10.333171 18970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "85b1f472cf6542e5b293da4b59da172c"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "067eeffb9db11508f44c2b677f46aa8c"
server_key_iv: "ba273b6b17b0995a1d312ac2bbfb9722"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.333248 18970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:10.344324 18970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:10.344666 18970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:10.344836 18970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:10.348795 18970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44253
I20260925 13:16:10.348836 19036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44253 every 8 connection(s)
I20260925 13:16:10.349208 18970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:10.350440 19037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.352732 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c: Bootstrap starting.
I20260925 13:16:10.353298 19037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.353539 19037 log.cc:824] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.354255 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c: No bootstrap required, opened a new log
I20260925 13:16:10.355809 19037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1f472cf6542e5b293da4b59da172c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260925 13:16:10.355949 19037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.355971 19037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b1f472cf6542e5b293da4b59da172c, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.356077 19037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1f472cf6542e5b293da4b59da172c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260925 13:16:10.356168 19037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:10.356208 19037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:10.356262 19037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.356784 19037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1f472cf6542e5b293da4b59da172c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260925 13:16:10.356909 19037 leader_election.cc:302] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b1f472cf6542e5b293da4b59da172c; no voters:
I20260925 13:16:10.357093 19037 leader_election.cc:288] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:10.357153 19040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:10.357365 19040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [term 1 LEADER]: Becoming Leader. State: Replica: 85b1f472cf6542e5b293da4b59da172c, State: Running, Role: LEADER
I20260925 13:16:10.357366 19037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:10.357479 19040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1f472cf6542e5b293da4b59da172c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260925 13:16:10.358028 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b1f472cf6542e5b293da4b59da172c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1f472cf6542e5b293da4b59da172c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } }
I20260925 13:16:10.358289 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:10.358193 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b1f472cf6542e5b293da4b59da172c. Latest consensus state: current_term: 1 leader_uuid: "85b1f472cf6542e5b293da4b59da172c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b1f472cf6542e5b293da4b59da172c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } }
I20260925 13:16:10.358495 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:10.358639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 18970
I20260925 13:16:10.358634 19050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:10.358755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:10.358973 392 external_mini_cluster.cc:1428] Setting key 2c54c5d1b79b3f22de66014d556c80a6
I20260925 13:16:10.359166 19050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:10.361002 19050 catalog_manager.cc:1347] Generated new cluster ID: e7526a31594d428495f2c885d219d7d6
I20260925 13:16:10.361055 19050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:10.366115 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.359473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.98.62:44253)
0925 13:16:10.360147 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.360154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.360342 (+ 188us) server_negotiation.cc:407] Connection header received
0925 13:16:10.360512 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.360518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.360572 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.360737 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:10.361009 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.362449 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.363616 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.364271 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.364740 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.365184 (+ 444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.365188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.365189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:10.365415 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.365457 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:10.365473 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.365762 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.365811 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.365894 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":58,"threads_started":1}
I20260925 13:16:10.370208 19050 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:10.370752 19050 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:10.377378 19050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c: Generated new TSK 0
I20260925 13:16:10.377641 19050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:10.380089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:10.483342 19061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:10.483578 19061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:10.483603 19061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:10.484887 19061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:10.484943 19061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:10.484972 19061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:16:10.486670 19061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:10.486969 19061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:10.487334 19061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:10.492659 19066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:10.492683 19069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.492810 19061 server_base.cc:1034] running on GCE node
W20260925 13:16:10.492808 19067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.493242 19061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:10.493525 19061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:10.494696 19061 hybrid_clock.cc:648] HybridClock initialized: now 1790342170494689 us; error 55 us; skew 500 ppm
I20260925 13:16:10.496372 19061 webserver.cc:460] Webserver started at http://127.0.98.1:39905/ using document root <none> and password file <none>
I20260925 13:16:10.496709 19061 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:10.496798 19061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:10.496961 19061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:10.498588 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "f44873b4df39447686dee1b6e924273b"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "22807f3e7cd4c88b8926156343c46638"
server_key_iv: "e6759c305396f21201c9e82c10c40168"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.499094 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f44873b4df39447686dee1b6e924273b"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "22807f3e7cd4c88b8926156343c46638"
server_key_iv: "e6759c305396f21201c9e82c10c40168"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.501044 19061 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 13:16:10.502154 19075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.502393 19061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:16:10.502494 19061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "f44873b4df39447686dee1b6e924273b"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "22807f3e7cd4c88b8926156343c46638"
server_key_iv: "e6759c305396f21201c9e82c10c40168"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.502588 19061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:10.538300 19061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:10.538666 19061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:10.538793 19061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:10.539031 19061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:10.539376 19061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:10.539413 19061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.539443 19061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:10.539458 19061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.546196 19061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46563
I20260925 13:16:10.546286 19188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46563 every 8 connection(s)
I20260925 13:16:10.546666 19061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:16:10.552529 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.547525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60871 (local address 127.0.98.62:44253)
0925 13:16:10.547641 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.547644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.547870 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:16:10.548396 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.548398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.548415 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.548480 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:10.549197 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.549843 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.550957 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.551168 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.551238 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.552226 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.552227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.552228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:10.552233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.552252 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:10.552256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.552310 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.552429 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.552460 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 13:16:10.552721 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.547494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.98.1:60871)
0925 13:16:10.547736 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.547775 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.548297 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.548540 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.548544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.548553 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:10.549094 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.549099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.549963 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.549964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.550852 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.550854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.551266 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.551267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.551387 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.552063 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:10.552064 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:10.552098 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:10.552126 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:10.552322 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:10.552326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:10.552329 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.552385 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.552481 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
I20260925 13:16:10.553313 19189 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44253
I20260925 13:16:10.553432 19189 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:10.553635 19189 heartbeater.cc:511] Master 127.0.98.62:44253 requested a full tablet report, sending...
I20260925 13:16:10.554304 19001 ts_manager.cc:194] Registered new tserver with Master: f44873b4df39447686dee1b6e924273b (127.0.98.1:46563)
I20260925 13:16:10.554982 19001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60871
I20260925 13:16:10.556718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19061
I20260925 13:16:10.556851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:16:10.557058 392 external_mini_cluster.cc:1428] Setting key 08aa551456fee2a1a30c3f4969ee4c12
I20260925 13:16:10.558092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:10.640686 19192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:10.640913 19192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:10.640937 19192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:10.642134 19192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:10.642175 19192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:10.642201 19192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:16:10.643514 19192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:10.643729 19192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:10.643999 19192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:10.649232 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.649441 19192 server_base.cc:1034] running on GCE node
W20260925 13:16:10.649277 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:10.649298 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.649879 19192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:10.650162 19192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:10.651333 19192 hybrid_clock.cc:648] HybridClock initialized: now 1790342170651327 us; error 47 us; skew 500 ppm
I20260925 13:16:10.652841 19192 webserver.cc:460] Webserver started at http://127.0.98.2:32783/ using document root <none> and password file <none>
I20260925 13:16:10.653113 19192 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:10.653184 19192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:10.653342 19192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:10.654539 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "0644c4f8131246119fb4d231e0855623"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "4d60d362508c3ad9f8d4867905d438eb"
server_key_iv: "bda2173beda54e885a908f502527e460"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.654932 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0644c4f8131246119fb4d231e0855623"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "4d60d362508c3ad9f8d4867905d438eb"
server_key_iv: "bda2173beda54e885a908f502527e460"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.656311 19192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:10.657094 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.657279 19192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:10.657377 19192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "0644c4f8131246119fb4d231e0855623"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "4d60d362508c3ad9f8d4867905d438eb"
server_key_iv: "bda2173beda54e885a908f502527e460"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.657457 19192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:10.678645 19192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:10.679010 19192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:10.679174 19192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:10.679430 19192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:10.679795 19192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:10.679852 19192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.679894 19192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:10.679940 19192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.685868 19192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42853
I20260925 13:16:10.685994 19319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42853 every 8 connection(s)
I20260925 13:16:10.686318 19192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:16:10.692034 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.687145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59589 (local address 127.0.98.62:44253)
0925 13:16:10.687244 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.687246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.687454 (+ 208us) server_negotiation.cc:407] Connection header received
0925 13:16:10.688004 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.688006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.688017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.688077 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:10.688814 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.689404 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.690602 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.690812 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.690871 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.691820 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.691821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.691821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:10.691826 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.691837 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:10.691840 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.691877 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.691944 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.691968 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 13:16:10.692286 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.687109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.98.2:59589)
0925 13:16:10.687321 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.687359 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.687884 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.688134 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.688137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.688145 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:10.688667 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.688680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.689530 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.689531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.690506 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.690510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.690899 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.690900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.691028 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.691682 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:10.691682 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:10.691715 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:10.691743 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:10.691880 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:10.691882 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:10.691885 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.691934 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.692021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
I20260925 13:16:10.692800 19320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44253
I20260925 13:16:10.692921 19320 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:10.693118 19320 heartbeater.cc:511] Master 127.0.98.62:44253 requested a full tablet report, sending...
I20260925 13:16:10.693401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19192
I20260925 13:16:10.693503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:16:10.693579 19001 ts_manager.cc:194] Registered new tserver with Master: 0644c4f8131246119fb4d231e0855623 (127.0.98.2:42853)
I20260925 13:16:10.693753 392 external_mini_cluster.cc:1428] Setting key 674af9487aa610f3d2feac532ffe12c1
I20260925 13:16:10.694113 19001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59589
I20260925 13:16:10.694903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:10.777035 19323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:10.777261 19323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:10.777283 19323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:10.778430 19323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:10.778468 19323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:10.778492 19323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:16:10.779771 19323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:10.779975 19323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:10.780246 19323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:10.785248 19328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:10.785236 19331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:10.785218 19329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:10.785849 19323 server_base.cc:1034] running on GCE node
I20260925 13:16:10.786084 19323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:10.786345 19323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:10.787487 19323 hybrid_clock.cc:648] HybridClock initialized: now 1790342170787464 us; error 36 us; skew 500 ppm
I20260925 13:16:10.788771 19323 webserver.cc:460] Webserver started at http://127.0.98.3:38263/ using document root <none> and password file <none>
I20260925 13:16:10.789034 19323 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:10.789079 19323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:10.789253 19323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:10.790447 19323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "b1e39ecc1c9b491e8765852c509da030"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "12e21dfeac863de41a5b84770590fe8a"
server_key_iv: "8e10b39c82dbf3bf4e28f0e23b95e987"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.790814 19323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b1e39ecc1c9b491e8765852c509da030"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "12e21dfeac863de41a5b84770590fe8a"
server_key_iv: "8e10b39c82dbf3bf4e28f0e23b95e987"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.792161 19323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:10.792922 19337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.793121 19323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:10.793215 19323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "b1e39ecc1c9b491e8765852c509da030"
format_stamp: "Formatted at 2026-09-25 13:16:10 on dist-test-slave-2d0s"
server_key: "12e21dfeac863de41a5b84770590fe8a"
server_key_iv: "8e10b39c82dbf3bf4e28f0e23b95e987"
server_key_version: "encryptionkey@0"
I20260925 13:16:10.793273 19323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:10.800696 19323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:10.800971 19323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:10.801077 19323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:10.801273 19323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:10.801548 19323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:10.801576 19323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.801599 19323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:10.801610 19323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:10.807366 19323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38755
I20260925 13:16:10.807462 19450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38755 every 8 connection(s)
I20260925 13:16:10.807775 19323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:16:10.809752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19323
I20260925 13:16:10.809882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:16:10.810127 392 external_mini_cluster.cc:1428] Setting key 38c837d486ac17ce3071ae5d2fbad4a0
I20260925 13:16:10.814015 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.808477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47213 (local address 127.0.98.62:44253)
0925 13:16:10.808546 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.808549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.808826 (+ 277us) server_negotiation.cc:407] Connection header received
0925 13:16:10.809307 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.809308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.809319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.809371 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:10.810487 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.811391 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.812607 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.812801 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.812857 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.813815 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.813816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.813816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:10.813821 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.813832 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:10.813836 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.813875 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.813926 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.813952 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:16:10.814255 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.808484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.98.3:47213)
0925 13:16:10.808707 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.808743 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.809223 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.809429 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.809432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.809441 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:10.810378 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.810384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.811497 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.811498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.812532 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.812535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.812879 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.812880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.812992 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.813610 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:10.813610 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:10.813696 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:10.813730 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:10.813876 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:10.813878 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:10.813881 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.813931 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.814020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260925 13:16:10.814690 19451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44253
I20260925 13:16:10.814817 19451 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:10.815016 19451 heartbeater.cc:511] Master 127.0.98.62:44253 requested a full tablet report, sending...
I20260925 13:16:10.815424 19000 ts_manager.cc:194] Registered new tserver with Master: b1e39ecc1c9b491e8765852c509da030 (127.0.98.3:38755)
I20260925 13:16:10.815771 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47213
I20260925 13:16:10.823045 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:16:10.827672 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.824508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.0.98.62:44253)
0925 13:16:10.824594 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.824597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.824701 (+ 104us) server_negotiation.cc:407] Connection header received
0925 13:16:10.824784 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.824785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.824800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.824868 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:10.825055 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.825555 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.826365 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.826562 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.826639 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.827369 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:10.827370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:10.827370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:10.827375 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:10.827390 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:10.827400 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:10.827499 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.827566 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.827577 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 13:16:10.829478 19000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:16:10.831295 19000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:16:10.838100 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.833824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.0.98.3:38755)
0925 13:16:10.834008 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.834013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.834040 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:16:10.834082 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.834085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.834101 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.834175 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.834429 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.835276 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.836766 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.837535 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.837604 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.837719 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.837940 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.837999 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260925 13:16:10.838073 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.833514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38755 (local address 127.0.0.1:58580)
0925 13:16:10.833937 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.833948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.834005 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.834244 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.834250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.834252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.834345 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.834347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.835368 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.835369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.836672 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.836674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.837754 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.837755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.837833 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.837845 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.837917 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.837967 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":6,"thread_start_us":75,"threads_started":1}
I20260925 13:16:10.838286 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.833744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.0.98.2:42853)
0925 13:16:10.833970 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.833974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.833986 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:16:10.834034 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.834038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.834067 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.834167 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.834455 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.835650 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.836981 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.837482 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.837754 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.837858 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.838146 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.838195 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260925 13:16:10.838415 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.833512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42853 (local address 127.0.0.1:42504)
0925 13:16:10.833756 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.833801 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.833887 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.834231 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.834252 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.834255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.834353 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.834355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.835778 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.835779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.836893 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.836895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.837870 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.837871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.837943 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.837948 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.838006 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.838065 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":36,"spinlock_wait_cycles":896,"thread_start_us":119,"threads_started":1}
I20260925 13:16:10.839313 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.833518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.0.1:50082)
0925 13:16:10.834015 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.834025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.834065 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.834275 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.834276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.834278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.834347 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.834350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.835792 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.835793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.838010 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.838013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.839088 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.839089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.839133 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.839137 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.839206 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.839250 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":11,"thread_start_us":93,"threads_started":1}
I20260925 13:16:10.839218 19385 tablet_service.cc:1467] Processing CreateTablet for tablet 0c381b77ecc2467ca460d9be6204628a (DEFAULT_TABLE table=test-table [id=685ca1af85404d66891e746ab736cf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:10.839361 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.833753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.0.98.1:46563)
0925 13:16:10.834108 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.834111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.834120 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:10.834172 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.834174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.834187 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.834264 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.834546 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.835670 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.838101 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.838915 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.838982 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.839058 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.839236 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.839277 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":91,"threads_started":1}
I20260925 13:16:10.839619 19385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c381b77ecc2467ca460d9be6204628a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.839905 19123 tablet_service.cc:1467] Processing CreateTablet for tablet 0c381b77ecc2467ca460d9be6204628a (DEFAULT_TABLE table=test-table [id=685ca1af85404d66891e746ab736cf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:10.840169 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c381b77ecc2467ca460d9be6204628a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.842262 19254 tablet_service.cc:1467] Processing CreateTablet for tablet 0c381b77ecc2467ca460d9be6204628a (DEFAULT_TABLE table=test-table [id=685ca1af85404d66891e746ab736cf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:10.842604 19254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c381b77ecc2467ca460d9be6204628a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:10.842842 19465 tablet_bootstrap.cc:492] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Bootstrap starting.
I20260925 13:16:10.843183 19466 tablet_bootstrap.cc:492] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: Bootstrap starting.
I20260925 13:16:10.843591 19465 tablet_bootstrap.cc:654] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.843832 19466 tablet_bootstrap.cc:654] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.843961 19465 log.cc:824] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.844153 19466 log.cc:824] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.844851 19465 tablet_bootstrap.cc:492] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: No bootstrap required, opened a new log
I20260925 13:16:10.844925 19465 ts_tablet_manager.cc:1397] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:10.844940 19466 tablet_bootstrap.cc:492] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: No bootstrap required, opened a new log
I20260925 13:16:10.845000 19466 ts_tablet_manager.cc:1397] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:10.846271 19469 tablet_bootstrap.cc:492] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Bootstrap starting.
I20260925 13:16:10.846923 19469 tablet_bootstrap.cc:654] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:10.847251 19469 log.cc:824] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:10.847182 19465 raft_consensus.cc:348] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.847337 19465 raft_consensus.cc:374] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.847402 19465 raft_consensus.cc:729] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e39ecc1c9b491e8765852c509da030, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.847594 19465 consensus_queue.cc:260] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.847926 19465 ts_tablet_manager.cc:1428] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:16:10.847945 19451 heartbeater.cc:503] Master 127.0.98.62:44253 was elected leader, sending a full tablet report...
I20260925 13:16:10.848028 19466 raft_consensus.cc:348] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.848187 19466 raft_consensus.cc:374] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.848217 19469 tablet_bootstrap.cc:492] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: No bootstrap required, opened a new log
I20260925 13:16:10.848224 19466 raft_consensus.cc:729] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44873b4df39447686dee1b6e924273b, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.848273 19469 ts_tablet_manager.cc:1397] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:10.848367 19466 consensus_queue.cc:260] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.848690 19189 heartbeater.cc:503] Master 127.0.98.62:44253 was elected leader, sending a full tablet report...
I20260925 13:16:10.848631 19466 ts_tablet_manager.cc:1428] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 13:16:10.850525 19469 raft_consensus.cc:348] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.850657 19469 raft_consensus.cc:374] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:10.850682 19469 raft_consensus.cc:729] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0644c4f8131246119fb4d231e0855623, State: Initialized, Role: FOLLOWER
I20260925 13:16:10.850811 19469 consensus_queue.cc:260] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.851056 19469 ts_tablet_manager.cc:1428] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:10.851158 19320 heartbeater.cc:503] Master 127.0.98.62:44253 was elected leader, sending a full tablet report...
W20260925 13:16:10.936923 19321 tablet.cc:2369] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:10.953186 19471 raft_consensus.cc:482] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:16:10.953334 19471 raft_consensus.cc:504] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.953809 19471 leader_election.cc:288] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0644c4f8131246119fb4d231e0855623 (127.0.98.2:42853), f44873b4df39447686dee1b6e924273b (127.0.98.1:46563)
I20260925 13:16:10.957948 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.954085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37541 (local address 127.0.98.2:42853)
0925 13:16:10.954145 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.954147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.954160 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:16:10.954249 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.954250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.954263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.954311 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.954501 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.955299 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.956882 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.957499 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.957589 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.957603 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.957815 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.957864 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:16:10.957954 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.954293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42993 (local address 127.0.98.1:46563)
0925 13:16:10.954342 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.954344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.954353 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:10.954456 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.954457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.954471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.954538 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:10.954682 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.955351 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.956881 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.957499 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.957622 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.957632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.957815 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.957864 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:16:10.957962 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.954018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42853 (local address 127.0.98.3:37541)
0925 13:16:10.954074 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.954098 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.954149 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.954361 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.954362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.954363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.954436 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.954438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.955434 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.955435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.956756 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.956759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.957596 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.957597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.957716 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.957720 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.957788 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.957864 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":13}
I20260925 13:16:10.957973 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.954098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.98.3:42993)
0925 13:16:10.954291 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:10.954302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:10.954363 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:10.954554 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:10.954555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:10.954556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:10.954605 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.954606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.955434 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.955435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.956756 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:10.956759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.957621 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.957622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:10.957739 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.957743 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:10.957795 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:16:10.957887 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":85,"thread_start_us":109,"threads_started":1}
I20260925 13:16:10.958359 19143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c381b77ecc2467ca460d9be6204628a" candidate_uuid: "b1e39ecc1c9b491e8765852c509da030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44873b4df39447686dee1b6e924273b" is_pre_election: true
I20260925 13:16:10.958359 19274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c381b77ecc2467ca460d9be6204628a" candidate_uuid: "b1e39ecc1c9b491e8765852c509da030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0644c4f8131246119fb4d231e0855623" is_pre_election: true
I20260925 13:16:10.958539 19274 raft_consensus.cc:2454] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e39ecc1c9b491e8765852c509da030 in term 0.
I20260925 13:16:10.958539 19143 raft_consensus.cc:2454] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e39ecc1c9b491e8765852c509da030 in term 0.
I20260925 13:16:10.958809 19338 leader_election.cc:302] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0644c4f8131246119fb4d231e0855623, b1e39ecc1c9b491e8765852c509da030; no voters:
I20260925 13:16:10.958994 19471 raft_consensus.cc:2781] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:16:10.959089 19471 raft_consensus.cc:482] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:16:10.959146 19471 raft_consensus.cc:3037] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.959868 19471 raft_consensus.cc:504] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.960034 19471 leader_election.cc:288] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [CANDIDATE]: Term 1 election: Requested vote from peers 0644c4f8131246119fb4d231e0855623 (127.0.98.2:42853), f44873b4df39447686dee1b6e924273b (127.0.98.1:46563)
I20260925 13:16:10.960227 19274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c381b77ecc2467ca460d9be6204628a" candidate_uuid: "b1e39ecc1c9b491e8765852c509da030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0644c4f8131246119fb4d231e0855623"
I20260925 13:16:10.960222 19143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c381b77ecc2467ca460d9be6204628a" candidate_uuid: "b1e39ecc1c9b491e8765852c509da030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44873b4df39447686dee1b6e924273b"
I20260925 13:16:10.960318 19143 raft_consensus.cc:3037] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.960340 19274 raft_consensus.cc:3037] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:10.961109 19274 raft_consensus.cc:2454] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e39ecc1c9b491e8765852c509da030 in term 1.
I20260925 13:16:10.961215 19143 raft_consensus.cc:2454] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e39ecc1c9b491e8765852c509da030 in term 1.
I20260925 13:16:10.961361 19338 leader_election.cc:302] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0644c4f8131246119fb4d231e0855623, b1e39ecc1c9b491e8765852c509da030; no voters:
I20260925 13:16:10.961458 19471 raft_consensus.cc:2781] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:10.961639 19471 raft_consensus.cc:686] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 1 LEADER]: Becoming Leader. State: Replica: b1e39ecc1c9b491e8765852c509da030, State: Running, Role: LEADER
I20260925 13:16:10.961802 19471 consensus_queue.cc:237] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260925 13:16:10.962653 19001 catalog_manager.cc:5495] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e39ecc1c9b491e8765852c509da030 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b1e39ecc1c9b491e8765852c509da030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e39ecc1c9b491e8765852c509da030" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } health_report { overall_health: HEALTHY } } }
I20260925 13:16:10.988175 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:10.985763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.0.98.3:38755)
0925 13:16:10.985817 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:10.985820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:10.985829 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:10.985872 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:10.985872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:10.985892 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:10.985973 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:10.986115 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.986579 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.987380 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:10.987578 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:10.987670 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:10.987827 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:10.987919 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:10.988033 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:10.988061 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 13:16:10.988069 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:16:10.990198 19274 raft_consensus.cc:1261] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 1 FOLLOWER]: Refusing update from remote peer b1e39ecc1c9b491e8765852c509da030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:10.990203 19143 raft_consensus.cc:1261] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 1 FOLLOWER]: Refusing update from remote peer b1e39ecc1c9b491e8765852c509da030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:10.990586 19471 consensus_queue.cc:1035] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0644c4f8131246119fb4d231e0855623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:10.990754 19475 consensus_queue.cc:1035] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44873b4df39447686dee1b6e924273b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:10.995657 19001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:10.995772 19001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:10.996594 19001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85b1f472cf6542e5b293da4b59da172c: Sending DeleteTablet for 3 replicas of tablet 0c381b77ecc2467ca460d9be6204628a
I20260925 13:16:10.996919 19123 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c381b77ecc2467ca460d9be6204628a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:10 UTC) from {username='slave'} at 127.0.0.1:50082
I20260925 13:16:10.996961 19254 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c381b77ecc2467ca460d9be6204628a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:10 UTC) from {username='slave'} at 127.0.0.1:42504
I20260925 13:16:10.997164 19488 tablet_replica.cc:330] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b: stopping tablet replica
I20260925 13:16:10.997188 19489 tablet_replica.cc:330] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: stopping tablet replica
I20260925 13:16:10.997241 19385 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c381b77ecc2467ca460d9be6204628a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:10 UTC) from {username='slave'} at 127.0.0.1:58580
I20260925 13:16:10.997277 19489 raft_consensus.cc:2229] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:10.997298 19488 raft_consensus.cc:2229] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:10.997375 19489 raft_consensus.cc:2258] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:10.997377 19488 raft_consensus.cc:2258] T 0c381b77ecc2467ca460d9be6204628a P f44873b4df39447686dee1b6e924273b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:10.997833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19061
I20260925 13:16:10.997921 19489 ts_tablet_manager.cc:1905] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:10.999758 19490 tablet_replica.cc:330] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: stopping tablet replica
I20260925 13:16:10.999881 19489 ts_tablet_manager.cc:1918] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:10.999960 19489 log.cc:1196] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/0c381b77ecc2467ca460d9be6204628a
I20260925 13:16:10.999980 19490 raft_consensus.cc:2229] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:16:11.000213 19489 ts_tablet_manager.cc:1939] T 0c381b77ecc2467ca460d9be6204628a P 0644c4f8131246119fb4d231e0855623: Deleting consensus metadata
I20260925 13:16:11.000293 19490 raft_consensus.cc:2258] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:11.000629 18985 catalog_manager.cc:4841] TS 0644c4f8131246119fb4d231e0855623 (127.0.98.2:42853): tablet 0c381b77ecc2467ca460d9be6204628a (table test-table [id=685ca1af85404d66891e746ab736cf28]) successfully deleted
I20260925 13:16:11.000864 19490 ts_tablet_manager.cc:1905] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:11.002485 19490 ts_tablet_manager.cc:1918] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:11.002565 19490 log.cc:1196] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/0c381b77ecc2467ca460d9be6204628a
I20260925 13:16:11.002826 19490 ts_tablet_manager.cc:1939] T 0c381b77ecc2467ca460d9be6204628a P b1e39ecc1c9b491e8765852c509da030: Deleting consensus metadata
W20260925 13:16:11.002954 18987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46563 (error 108)
I20260925 13:16:11.003324 18986 catalog_manager.cc:4841] TS b1e39ecc1c9b491e8765852c509da030 (127.0.98.3:38755): tablet 0c381b77ecc2467ca460d9be6204628a (table test-table [id=685ca1af85404d66891e746ab736cf28]) successfully deleted
W20260925 13:16:11.003470 19460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:11.003334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.0.1:50094)
0925 13:16:11.003382 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:11.003434 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46563: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 13:16:11.003608 18987 catalog_manager.cc:4584] TS f44873b4df39447686dee1b6e924273b (127.0.98.1:46563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c381b77ecc2467ca460d9be6204628a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46563: connect: Connection reset by peer (error 104)
I20260925 13:16:11.003736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19192
I20260925 13:16:11.008946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19323
I20260925 13:16:11.013713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 18970
2026-09-25T13:16:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (806 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:11 dist-test-slave-2d0s krb5kdc[19494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:11 dist-test-slave-2d0s krb5kdc[19494](info): set up 2 sockets
Sep 25 13:16:11 dist-test-slave-2d0s krb5kdc[19494](info): commencing operation
krb5kdc: starting...
W20260925 13:16:12.061829 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:12.189320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:12.273639 19507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:12.273972 19507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:12.274009 19507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:12.275295 19507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:12.275385 19507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:12.275409 19507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:12.275442 19507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:12.275494 19507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:12.277132 19507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:12.277456 19507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:12.277774 19507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:12.282752 19515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.282840 19513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.282752 19512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:12.282958 19507 server_base.cc:1034] running on GCE node
I20260925 13:16:12.283229 19507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:12.283517 19507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:12.284663 19507 hybrid_clock.cc:648] HybridClock initialized: now 1790342172284651 us; error 32 us; skew 500 ppm
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:12.287478 19507 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:12.287940 19507 webserver.cc:460] Webserver started at http://127.0.98.62:36151/ using document root <none> and password file <none>
I20260925 13:16:12.288200 19507 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:12.288249 19507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:12.288384 19507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:12.289508 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "64f3cbd96a0fab7a78bfc14107a866bd"
server_key_iv: "f365eba75c132fb51bf1b7c5499b96d5"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.289963 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "64f3cbd96a0fab7a78bfc14107a866bd"
server_key_iv: "f365eba75c132fb51bf1b7c5499b96d5"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.291397 19507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:12.292438 19522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.292651 19507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:12.292769 19507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "64f3cbd96a0fab7a78bfc14107a866bd"
server_key_iv: "f365eba75c132fb51bf1b7c5499b96d5"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.292845 19507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:12.310998 19507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:12.313900 19507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:12.314090 19507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:12.318233 19507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40543
I20260925 13:16:12.318279 19574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40543 every 8 connection(s)
I20260925 13:16:12.318678 19507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:12.319702 19575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:12.322235 19575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade: Bootstrap starting.
I20260925 13:16:12.323297 19575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:12.323650 19575 log.cc:824] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:12.324739 19575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade: No bootstrap required, opened a new log
I20260925 13:16:12.324769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19507
I20260925 13:16:12.324950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:12.325181 392 external_mini_cluster.cc:1428] Setting key 4ed9e1f3402581505295eb6b2d824c97
I20260925 13:16:12.327385 19575 raft_consensus.cc:348] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260925 13:16:12.327607 19575 raft_consensus.cc:374] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:12.327652 19575 raft_consensus.cc:729] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d27a85c0ddfe4e99b0ad19dc5c1b9ade, State: Initialized, Role: FOLLOWER
I20260925 13:16:12.327818 19575 consensus_queue.cc:260] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260925 13:16:12.327904 19575 raft_consensus.cc:388] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:12.327929 19575 raft_consensus.cc:482] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:12.327986 19575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:12.329293 19575 raft_consensus.cc:504] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260925 13:16:12.329422 19575 leader_election.cc:302] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d27a85c0ddfe4e99b0ad19dc5c1b9ade; no voters:
I20260925 13:16:12.329603 19575 leader_election.cc:288] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:12.329707 19580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:12.329908 19575 sys_catalog.cc:564] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:12.329910 19580 raft_consensus.cc:686] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [term 1 LEADER]: Becoming Leader. State: Replica: d27a85c0ddfe4e99b0ad19dc5c1b9ade, State: Running, Role: LEADER
I20260925 13:16:12.330044 19580 consensus_queue.cc:237] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260925 13:16:12.330494 19582 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader d27a85c0ddfe4e99b0ad19dc5c1b9ade. Latest consensus state: current_term: 1 leader_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } } }
I20260925 13:16:12.330608 19582 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:12.330829 19589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:12.331370 19589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:12.331763 19581 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27a85c0ddfe4e99b0ad19dc5c1b9ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } } }
I20260925 13:16:12.331938 19581 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:12.333850 19589 catalog_manager.cc:1347] Generated new cluster ID: 23d95b3843ab4084b712cb08eb2647b9
I20260925 13:16:12.333909 19589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:12.340212 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.325915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.0.98.62:40543)
0925 13:16:12.326170 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.326181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.326217 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:16:12.327328 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.327339 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.327417 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.327600 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:12.328528 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.330071 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.331326 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.332025 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.332161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.335403 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:12.335407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:12.335408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:12.335422 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:12.337615 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.338165 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.338169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.338169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.338842 (+ 673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.339126 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.339127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.339127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.339497 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:12.339655 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.339805 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.339960 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
I20260925 13:16:12.353027 19589 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:12.353739 19589 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:12.359884 19589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade: Generated new TSK 0
I20260925 13:16:12.360133 19589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:16:12.397459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:12.484802 19601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:12.485059 19601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:12.485087 19601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:12.486392 19601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:12.486434 19601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:12.486465 19601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:16:12.488089 19601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:12.488385 19601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:12.488605 19601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:12.493971 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.493986 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.493990 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:12.494294 19601 server_base.cc:1034] running on GCE node
I20260925 13:16:12.494576 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:12.494877 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:12.496045 19601 hybrid_clock.cc:648] HybridClock initialized: now 1790342172496033 us; error 31 us; skew 500 ppm
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:12.498881 19601 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:16:12.499362 19601 webserver.cc:460] Webserver started at http://127.0.98.1:36625/ using document root <none> and password file <none>
I20260925 13:16:12.499605 19601 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:12.499676 19601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:12.499827 19601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:12.500942 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "0068e919d1404bd79e03dbd7bb77b320"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "f9cefc1cf649d2eafabc6e23ce9491a0"
server_key_iv: "a3f1384282dc37a94b122c43650382d5"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.501315 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0068e919d1404bd79e03dbd7bb77b320"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "f9cefc1cf649d2eafabc6e23ce9491a0"
server_key_iv: "a3f1384282dc37a94b122c43650382d5"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.502791 19601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:16:12.503566 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.503778 19601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:12.503885 19601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "0068e919d1404bd79e03dbd7bb77b320"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "f9cefc1cf649d2eafabc6e23ce9491a0"
server_key_iv: "a3f1384282dc37a94b122c43650382d5"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.503962 19601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:12.519605 19601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:12.522173 19601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:12.522356 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:12.522620 19601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:12.522951 19601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:12.523008 19601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.523056 19601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:12.523082 19601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.529451 19601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35405
I20260925 13:16:12.529489 19729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35405 every 8 connection(s)
I20260925 13:16:12.529927 19601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:16:12.533345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19601
I20260925 13:16:12.533485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:16:12.533736 392 external_mini_cluster.cc:1428] Setting key d3e4d636dc63f8c0d0964409e4bebb8a
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:12.541085 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.530768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44465 (local address 127.0.98.62:40543)
0925 13:16:12.530813 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.530815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.531055 (+ 240us) server_negotiation.cc:407] Connection header received
0925 13:16:12.532154 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.532155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.532172 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.532230 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:12.532907 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.533504 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.534852 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.535091 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.535179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.537792 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:12.537793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:12.537794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:12.537807 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:12.539581 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.540022 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.540023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.540024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.540094 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.540339 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.540340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.540340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.540510 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:12.540585 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.540849 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.540964 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:16:12.541114 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.530736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.1:44465)
0925 13:16:12.530878 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:12.530927 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:12.532035 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:12.532313 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:12.532317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:12.532602 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:12.532804 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.532808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.533587 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.533589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.534761 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.534764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.535192 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.535193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.535337 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.535949 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:12.535960 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:12.537636 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:12.539683 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:12.539691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:12.539694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:12.539944 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:12.540166 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:12.540167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:12.540167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:12.540272 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:12.540580 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:12.540584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:12.540715 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 13:16:12.540799 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:16:12.540936 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260925 13:16:12.541718 19730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260925 13:16:12.541836 19730 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:12.542075 19730 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:16:12.542733 19539 ts_manager.cc:194] Registered new tserver with Master: 0068e919d1404bd79e03dbd7bb77b320 (127.0.98.1:35405)
I20260925 13:16:12.543397 19539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:16:12.560479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:12.645701 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:12.645967 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:12.646008 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:12.647225 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:12.647300 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:12.647347 19735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:16:12.648934 19735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:12.649226 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:12.649469 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:12.654659 19740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.654632 19741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.654799 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:12.654906 19735 server_base.cc:1034] running on GCE node
I20260925 13:16:12.655143 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:12.655397 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:12.656579 19735 hybrid_clock.cc:648] HybridClock initialized: now 1790342172656574 us; error 53 us; skew 500 ppm
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:12.659245 19735 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:16:12.659744 19735 webserver.cc:460] Webserver started at http://127.0.98.2:42045/ using document root <none> and password file <none>
I20260925 13:16:12.659982 19735 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:12.660053 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:12.660200 19735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:12.661289 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b79bfaa1257469b98a5a565cb09bfe3"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "295d1e6e349b529111c37904506920de"
server_key_iv: "4d23587926c17ac3c3030206af8beca2"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.661710 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b79bfaa1257469b98a5a565cb09bfe3"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "295d1e6e349b529111c37904506920de"
server_key_iv: "4d23587926c17ac3c3030206af8beca2"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.663060 19735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:12.663803 19750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.663976 19735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.664073 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "9b79bfaa1257469b98a5a565cb09bfe3"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "295d1e6e349b529111c37904506920de"
server_key_iv: "4d23587926c17ac3c3030206af8beca2"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.664156 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:12.676978 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:12.678923 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:12.679088 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:12.679342 19735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:12.679656 19735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:12.679709 19735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.679752 19735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:12.679778 19735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.685634 19735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39235
I20260925 13:16:12.685716 19863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39235 every 8 connection(s)
I20260925 13:16:12.686031 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:12.695868 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.686786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47295 (local address 127.0.98.62:40543)
0925 13:16:12.686874 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.686876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.687044 (+ 168us) server_negotiation.cc:407] Connection header received
0925 13:16:12.687986 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.687987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.687997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.688050 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:12.688686 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.689205 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.690244 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.690478 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.690546 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.692712 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:12.692714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:12.692714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:12.692723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:12.694295 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.694761 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.694763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.694763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.694830 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.695121 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.695122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.695122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.695249 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:12.695330 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.695640 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.695738 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 13:16:12.696051 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.686727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.2:47295)
0925 13:16:12.686915 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:12.686953 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:12.687881 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:12.688107 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:12.688111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:12.688374 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:12.688595 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.688599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.689296 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.689297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.690136 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.690138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.690558 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.690559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.690676 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.691154 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:12.691163 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:12.692603 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:12.694406 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:12.694421 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:12.694423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:12.694664 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:12.694903 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:12.694904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:12.694904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:12.695045 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:12.695333 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:12.695341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:12.695494 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 13:16:12.695618 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 13:16:12.695781 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260925 13:16:12.696072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19735
I20260925 13:16:12.696201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:16:12.696458 392 external_mini_cluster.cc:1428] Setting key 037734441eb178bb3be9532e7a430af4
I20260925 13:16:12.696700 19864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260925 13:16:12.696812 19864 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:12.697008 19864 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260925 13:16:12.697428 19539 ts_manager.cc:194] Registered new tserver with Master: 9b79bfaa1257469b98a5a565cb09bfe3 (127.0.98.2:39235)
I20260925 13:16:12.697886 19539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47295
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:16:12.720567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:12.804111 19869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:12.804378 19869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:12.804418 19869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:12.805637 19869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:12.805749 19869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:12.805799 19869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:16:12.807468 19869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:12.807767 19869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:12.808024 19869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:12.813310 19877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.813310 19874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:12.813324 19875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:12.813601 19869 server_base.cc:1034] running on GCE node
I20260925 13:16:12.813856 19869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:12.814131 19869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:12.815284 19869 hybrid_clock.cc:648] HybridClock initialized: now 1790342172815271 us; error 30 us; skew 500 ppm
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:12.817983 19869 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:16:12.818480 19869 webserver.cc:460] Webserver started at http://127.0.98.3:43981/ using document root <none> and password file <none>
I20260925 13:16:12.818747 19869 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:12.818797 19869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:12.818961 19869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:12.820091 19869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "6596a9adfe34fabd70cbab2994fdfb12"
server_key_iv: "fa9016009bb6d858cce1ec3aea93d4fe"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.820474 19869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "6596a9adfe34fabd70cbab2994fdfb12"
server_key_iv: "fa9016009bb6d858cce1ec3aea93d4fe"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.822019 19869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:16:12.822836 19884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.823027 19869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:12.823125 19869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f"
format_stamp: "Formatted at 2026-09-25 13:16:12 on dist-test-slave-2d0s"
server_key: "6596a9adfe34fabd70cbab2994fdfb12"
server_key_iv: "fa9016009bb6d858cce1ec3aea93d4fe"
server_key_version: "encryptionkey@0"
I20260925 13:16:12.823213 19869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:12.838083 19869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:12.840273 19869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:12.840453 19869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:12.840718 19869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:12.841041 19869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:12.841094 19869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.841138 19869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:12.841163 19869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:12.847793 19869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46783
I20260925 13:16:12.847833 19997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46783 every 8 connection(s)
I20260925 13:16:12.848174 19869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:16:12 dist-test-slave-2d0s krb5kdc[19494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:12.856011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 19869
I20260925 13:16:12.856137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:16:12.856333 392 external_mini_cluster.cc:1428] Setting key 4fbc8387d41ed0975ae18103bed7d138
I20260925 13:16:12.858104 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.848904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36179 (local address 127.0.98.62:40543)
0925 13:16:12.848976 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.848978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.849217 (+ 239us) server_negotiation.cc:407] Connection header received
0925 13:16:12.850262 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.850263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.850274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.850329 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:12.850924 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.851434 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.852477 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.852677 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.852743 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.854938 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:12.854939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:12.854939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:12.854946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:12.856605 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.857130 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.857131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.857132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.857201 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.857478 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.857478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.857478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.857599 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:12.857652 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.857969 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.858039 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 13:16:12.858304 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.848894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.3:36179)
0925 13:16:12.849086 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:12.849124 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:12.850162 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:12.850387 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:12.850391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:12.850653 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:12.850853 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.850857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.851540 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.851541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.852377 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.852379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.852757 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.852757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.852856 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.853390 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:12.853398 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:12.854851 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:12.856694 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:12.856702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:12.856706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:12.857057 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:12.857290 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:12.857291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:12.857291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:12.857418 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:12.857657 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:12.857700 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:12.857849 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 13:16:12.857937 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 13:16:12.858079 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260925 13:16:12.858829 19998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260925 13:16:12.858938 19998 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:12.859131 19998 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260925 13:16:12.859532 19539 ts_manager.cc:194] Registered new tserver with Master: 69acd65ae95f4dd9a24e1f03ed3f0a9f (127.0.98.3:46783)
I20260925 13:16:12.860047 19539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36179
I20260925 13:16:12.868319 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:16:12.875871 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.869796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.0.98.62:40543)
0925 13:16:12.869872 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.869874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.869942 (+ 68us) server_negotiation.cc:407] Connection header received
0925 13:16:12.870012 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.870013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.870024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.870073 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:12.870634 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.871124 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.871831 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.872006 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.872053 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.873259 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:12.873260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:12.873261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:12.873269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:12.874791 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.875146 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.875147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.875147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.875200 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:12.875370 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:12.875371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:12.875371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:12.875516 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:12.875591 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.875702 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.875765 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:16:12.877604 19539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:16:12.879335 19539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:16:12.886308 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.882160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.0.98.1:35405)
0925 13:16:12.882539 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.882544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.882553 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:12.882620 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.882622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.882639 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.882732 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:12.882881 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.883919 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.885310 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.885946 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.886019 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.886076 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.886185 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.886230 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":71,"threads_started":1}
I20260925 13:16:12.886327 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.881900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35405 (local address 127.0.0.1:53342)
0925 13:16:12.882381 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:12.882390 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:12.882440 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:12.882731 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:12.882733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:12.882735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:12.882785 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.882787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.884019 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.884020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.885203 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.885206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.886028 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.886030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.886104 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.886109 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:16:12.886169 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:16:12.886216 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":28,"thread_start_us":81,"threads_started":1}
I20260925 13:16:12.886518 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.882126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.98.3:46783)
0925 13:16:12.882317 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.882322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.882333 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:16:12.882377 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.882379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.882394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.882969 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:12.883050 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.883701 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.885297 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.886041 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.886105 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.886160 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.886381 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.886425 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260925 13:16:12.886598 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.881891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46783 (local address 127.0.0.1:50850)
0925 13:16:12.882262 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:12.882271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:12.882310 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:12.882845 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:12.882846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:12.882848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:12.882892 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.882895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.883813 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.883814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.885217 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.885219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.886194 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.886195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.886246 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.886250 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:12.886295 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:16:12.886325 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":51,"thread_start_us":91,"threads_started":1}
I20260925 13:16:12.886902 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.881986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.0.98.2:39235)
0925 13:16:12.882183 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:12.882188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:12.882217 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:16:12.882266 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:12.882268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:12.882289 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:12.882370 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:12.882556 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.883697 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.885059 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.885804 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.885884 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.885948 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:12.886267 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 13:16:12.886313 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
I20260925 13:16:12.887033 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:12.881871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39235 (local address 127.0.0.1:32902)
0925 13:16:12.882051 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:12.882095 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:12.882165 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:12.882379 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:12.882383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:12.882385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:12.882473 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.882476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.883811 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.883812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.884969 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:12.884972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:12.885903 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:12.885904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:12.886003 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:12.886021 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 13:16:12.886102 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:16:12.886163 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":71,"thread_start_us":62,"threads_started":1}
I20260925 13:16:12.887327 19664 tablet_service.cc:1467] Processing CreateTablet for tablet 3625939dbf2842b7afed5d3c1ed15ce0 (DEFAULT_TABLE table=test-table [id=804e8c5001a04282a850545672718742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:12.887341 19932 tablet_service.cc:1467] Processing CreateTablet for tablet 3625939dbf2842b7afed5d3c1ed15ce0 (DEFAULT_TABLE table=test-table [id=804e8c5001a04282a850545672718742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:12.887667 19664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3625939dbf2842b7afed5d3c1ed15ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:12.887686 19932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3625939dbf2842b7afed5d3c1ed15ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:12.887701 19798 tablet_service.cc:1467] Processing CreateTablet for tablet 3625939dbf2842b7afed5d3c1ed15ce0 (DEFAULT_TABLE table=test-table [id=804e8c5001a04282a850545672718742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:12.887988 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3625939dbf2842b7afed5d3c1ed15ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:12.890640 20015 tablet_bootstrap.cc:492] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: Bootstrap starting.
I20260925 13:16:12.891232 20016 tablet_bootstrap.cc:492] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Bootstrap starting.
I20260925 13:16:12.891304 20017 tablet_bootstrap.cc:492] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Bootstrap starting.
I20260925 13:16:12.891372 20015 tablet_bootstrap.cc:654] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:12.891866 20017 tablet_bootstrap.cc:654] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:12.891952 20015 log.cc:824] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:12.892074 20016 tablet_bootstrap.cc:654] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:12.892498 20017 log.cc:824] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:12.892611 20016 log.cc:824] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:12.893141 20015 tablet_bootstrap.cc:492] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: No bootstrap required, opened a new log
I20260925 13:16:12.893200 20015 ts_tablet_manager.cc:1397] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:16:12.893195 20017 tablet_bootstrap.cc:492] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: No bootstrap required, opened a new log
I20260925 13:16:12.893257 20017 ts_tablet_manager.cc:1397] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:12.894155 20016 tablet_bootstrap.cc:492] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: No bootstrap required, opened a new log
I20260925 13:16:12.894230 20016 ts_tablet_manager.cc:1397] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:16:12.894902 20017 raft_consensus.cc:348] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:12.895077 20017 raft_consensus.cc:374] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:12.895120 20017 raft_consensus.cc:729] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b79bfaa1257469b98a5a565cb09bfe3, State: Initialized, Role: FOLLOWER
I20260925 13:16:12.895244 20017 consensus_queue.cc:260] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:12.895278 20015 raft_consensus.cc:348] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:12.895457 20015 raft_consensus.cc:374] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:12.895504 20015 raft_consensus.cc:729] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0068e919d1404bd79e03dbd7bb77b320, State: Initialized, Role: FOLLOWER
I20260925 13:16:12.895562 20017 ts_tablet_manager.cc:1428] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:16:12.895648 20015 consensus_queue.cc:260] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:12.895726 19864 heartbeater.cc:503] Master 127.0.98.62:40543 was elected leader, sending a full tablet report...
I20260925 13:16:12.895934 20015 ts_tablet_manager.cc:1428] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:12.895951 19730 heartbeater.cc:503] Master 127.0.98.62:40543 was elected leader, sending a full tablet report...
I20260925 13:16:12.896588 20016 raft_consensus.cc:348] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:12.896772 20016 raft_consensus.cc:374] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:12.896811 20016 raft_consensus.cc:729] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69acd65ae95f4dd9a24e1f03ed3f0a9f, State: Initialized, Role: FOLLOWER
I20260925 13:16:12.896919 20016 consensus_queue.cc:260] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:12.897198 20016 ts_tablet_manager.cc:1428] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:12.897221 19998 heartbeater.cc:503] Master 127.0.98.62:40543 was elected leader, sending a full tablet report...
W20260925 13:16:12.936589 19865 tablet.cc:2369] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:16:13.030793 19731 tablet.cc:2369] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:16:13.098744 19999 tablet.cc:2369] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:13.109470 20021 raft_consensus.cc:482] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:16:13.109642 20021 raft_consensus.cc:504] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:13.110117 20021 leader_election.cc:288] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69acd65ae95f4dd9a24e1f03ed3f0a9f (127.0.98.3:46783), 0068e919d1404bd79e03dbd7bb77b320 (127.0.98.1:35405)
I20260925 13:16:13.113407 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.110309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46783 (local address 127.0.98.2:59291)
0925 13:16:13.110372 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.110385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.110432 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.110615 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.110616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.110618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.110700 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.110702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.111370 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.111371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.112530 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.112533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.113123 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.113125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.113214 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.113218 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.113268 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.113311 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 13:16:13.113410 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.110395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59291 (local address 127.0.98.3:46783)
0925 13:16:13.110449 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.110451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.110462 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:16:13.110518 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.110519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.110530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.110590 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.110774 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.111290 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.112619 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.113039 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.113094 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.113102 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.113292 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.113336 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:16:13.113510 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.110424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45983 (local address 127.0.98.1:35405)
0925 13:16:13.110497 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.110499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.110602 (+ 103us) server_negotiation.cc:407] Connection header received
0925 13:16:13.110632 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.110633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.110644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.110708 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.110893 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.111373 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.112726 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.113120 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.113191 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.113203 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.113387 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.113436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:16:13.113569 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.110336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35405 (local address 127.0.98.2:45983)
0925 13:16:13.110508 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.110518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.110582 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.110721 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.110722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.110723 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.110815 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.110817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.111499 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.111500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.112641 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.112643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.113215 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.113216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.113308 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.113313 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.113368 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.113405 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260925 13:16:13.113826 19952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3625939dbf2842b7afed5d3c1ed15ce0" candidate_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" is_pre_election: true
I20260925 13:16:13.113893 19684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3625939dbf2842b7afed5d3c1ed15ce0" candidate_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0068e919d1404bd79e03dbd7bb77b320" is_pre_election: true
I20260925 13:16:13.113998 19952 raft_consensus.cc:2454] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b79bfaa1257469b98a5a565cb09bfe3 in term 0.
I20260925 13:16:13.114075 19684 raft_consensus.cc:2454] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b79bfaa1257469b98a5a565cb09bfe3 in term 0.
I20260925 13:16:13.114220 19751 leader_election.cc:302] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69acd65ae95f4dd9a24e1f03ed3f0a9f, 9b79bfaa1257469b98a5a565cb09bfe3; no voters:
I20260925 13:16:13.114353 20021 raft_consensus.cc:2781] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:16:13.114423 20021 raft_consensus.cc:482] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:16:13.114440 20021 raft_consensus.cc:3037] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.115126 20021 raft_consensus.cc:504] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:13.115278 20021 leader_election.cc:288] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [CANDIDATE]: Term 1 election: Requested vote from peers 69acd65ae95f4dd9a24e1f03ed3f0a9f (127.0.98.3:46783), 0068e919d1404bd79e03dbd7bb77b320 (127.0.98.1:35405)
I20260925 13:16:13.115429 19952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3625939dbf2842b7afed5d3c1ed15ce0" candidate_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f"
I20260925 13:16:13.115461 19684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3625939dbf2842b7afed5d3c1ed15ce0" candidate_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0068e919d1404bd79e03dbd7bb77b320"
I20260925 13:16:13.115531 19684 raft_consensus.cc:3037] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.115545 19952 raft_consensus.cc:3037] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.116271 19952 raft_consensus.cc:2454] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b79bfaa1257469b98a5a565cb09bfe3 in term 1.
I20260925 13:16:13.116277 19684 raft_consensus.cc:2454] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b79bfaa1257469b98a5a565cb09bfe3 in term 1.
I20260925 13:16:13.116446 19751 leader_election.cc:302] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69acd65ae95f4dd9a24e1f03ed3f0a9f, 9b79bfaa1257469b98a5a565cb09bfe3; no voters:
I20260925 13:16:13.116557 20021 raft_consensus.cc:2781] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:13.116704 20021 raft_consensus.cc:686] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 1 LEADER]: Becoming Leader. State: Replica: 9b79bfaa1257469b98a5a565cb09bfe3, State: Running, Role: LEADER
I20260925 13:16:13.116817 20021 consensus_queue.cc:237] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } }
I20260925 13:16:13.117975 19537 catalog_manager.cc:5495] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b79bfaa1257469b98a5a565cb09bfe3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b79bfaa1257469b98a5a565cb09bfe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 } health_report { overall_health: UNKNOWN } } }
I20260925 13:16:13.172585 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.170189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.0.98.2:39235)
0925 13:16:13.170240 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.170242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.170253 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:16:13.170303 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.170304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.170324 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.170407 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:13.170526 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.171007 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.171783 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.171982 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.172069 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.172216 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:13.172320 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:13.172415 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.172435 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.172457 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:16:13.174391 19952 raft_consensus.cc:1261] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 1 FOLLOWER]: Refusing update from remote peer 9b79bfaa1257469b98a5a565cb09bfe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:13.174412 19684 raft_consensus.cc:1261] T 3625939dbf2842b7afed5d3c1ed15ce0 P 0068e919d1404bd79e03dbd7bb77b320 [term 1 FOLLOWER]: Refusing update from remote peer 9b79bfaa1257469b98a5a565cb09bfe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:13.174791 20021 consensus_queue.cc:1035] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69acd65ae95f4dd9a24e1f03ed3f0a9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:13.174934 20025 consensus_queue.cc:1035] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0068e919d1404bd79e03dbd7bb77b320" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:13.177541 20028 mvcc.cc:205] Tried to move back new op lower bound from 7333241541319819264 to 7333241541087207424. Current Snapshot: MvccSnapshot[applied={T|T < 7333241541319819264}]
I20260925 13:16:13.177644 20030 mvcc.cc:205] Tried to move back new op lower bound from 7333241541319819264 to 7333241541087207424. Current Snapshot: MvccSnapshot[applied={T|T < 7333241541319819264 or (T in {7333241541319819264})}]
I20260925 13:16:13.177692 20029 mvcc.cc:205] Tried to move back new op lower bound from 7333241541319819264 to 7333241541087207424. Current Snapshot: MvccSnapshot[applied={T|T < 7333241541319819264}]
I20260925 13:16:13.179932 19537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:13.180035 19537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:13.180828 19537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d27a85c0ddfe4e99b0ad19dc5c1b9ade: Sending DeleteTablet for 3 replicas of tablet 3625939dbf2842b7afed5d3c1ed15ce0
I20260925 13:16:13.181237 19664 tablet_service.cc:1514] Processing DeleteTablet for tablet 3625939dbf2842b7afed5d3c1ed15ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53342
I20260925 13:16:13.181454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19601
I20260925 13:16:13.181483 19932 tablet_service.cc:1514] Processing DeleteTablet for tablet 3625939dbf2842b7afed5d3c1ed15ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50850
I20260925 13:16:13.182974 19798 tablet_service.cc:1514] Processing DeleteTablet for tablet 3625939dbf2842b7afed5d3c1ed15ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32902
I20260925 13:16:13.183044 20038 tablet_replica.cc:330] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: stopping tablet replica
I20260925 13:16:13.183157 20038 raft_consensus.cc:2229] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:13.183200 20039 tablet_replica.cc:330] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: stopping tablet replica
I20260925 13:16:13.183280 20038 raft_consensus.cc:2258] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:13.183295 20039 raft_consensus.cc:2229] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:16:13.183393 20039 raft_consensus.cc:2258] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:13.183799 20039 ts_tablet_manager.cc:1905] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:13.185328 20039 ts_tablet_manager.cc:1918] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:13.185420 20039 log.cc:1196] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/3625939dbf2842b7afed5d3c1ed15ce0
I20260925 13:16:13.185710 20039 ts_tablet_manager.cc:1939] T 3625939dbf2842b7afed5d3c1ed15ce0 P 9b79bfaa1257469b98a5a565cb09bfe3: Deleting consensus metadata
I20260925 13:16:13.186133 20038 ts_tablet_manager.cc:1905] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:13.186151 19524 catalog_manager.cc:4841] TS 9b79bfaa1257469b98a5a565cb09bfe3 (127.0.98.2:39235): tablet 3625939dbf2842b7afed5d3c1ed15ce0 (table test-table [id=804e8c5001a04282a850545672718742]) successfully deleted
I20260925 13:16:13.187433 20038 ts_tablet_manager.cc:1918] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:13.187531 20038 log.cc:1196] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/3625939dbf2842b7afed5d3c1ed15ce0
W20260925 13:16:13.187767 19525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35405 (error 108)
I20260925 13:16:13.187844 20038 ts_tablet_manager.cc:1939] T 3625939dbf2842b7afed5d3c1ed15ce0 P 69acd65ae95f4dd9a24e1f03ed3f0a9f: Deleting consensus metadata
I20260925 13:16:13.188304 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.188181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35405 (local address 127.0.0.1:53346)
0925 13:16:13.188227 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.188264 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:16:13.188324 19523 catalog_manager.cc:4841] TS 69acd65ae95f4dd9a24e1f03ed3f0a9f (127.0.98.3:46783): tablet 3625939dbf2842b7afed5d3c1ed15ce0 (table test-table [id=804e8c5001a04282a850545672718742]) successfully deleted
W20260925 13:16:13.188433 19525 catalog_manager.cc:4584] TS 0068e919d1404bd79e03dbd7bb77b320 (127.0.98.1:35405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3625939dbf2842b7afed5d3c1ed15ce0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35405: connect: Connection refused (error 111)
I20260925 13:16:13.188537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19735
I20260925 13:16:13.193127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19869
I20260925 13:16:13.197335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 19507
2026-09-25T13:16:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T13:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:13Z Disabled control of system clock
I20260925 13:16:13.225268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46775
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:13.309134 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:13.309401 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:13.309448 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:13.310719 20044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:13.310791 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:13.310815 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:13.310845 20044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:13.310873 20044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:13.312249 20044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:13.312570 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:13.312944 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:13.317865 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.317879 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.317881 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:13.318174 20044 server_base.cc:1034] running on GCE node
I20260925 13:16:13.318377 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:13.318683 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:13.319859 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790342173319844 us; error 29 us; skew 500 ppm
I20260925 13:16:13.321110 20044 webserver.cc:460] Webserver started at http://127.0.98.62:35805/ using document root <none> and password file <none>
I20260925 13:16:13.321354 20044 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:13.321425 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:13.321586 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:13.322733 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "1e429f271d9a4604abd42001260c679f"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "889bfb19cc0997df34ed99cde17b5879"
server_key_iv: "75ca701b8d45af6ea45cdf406e5cfe63"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.323114 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e429f271d9a4604abd42001260c679f"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "889bfb19cc0997df34ed99cde17b5879"
server_key_iv: "75ca701b8d45af6ea45cdf406e5cfe63"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.324496 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:13.325246 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.325417 20044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:13.325522 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "1e429f271d9a4604abd42001260c679f"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "889bfb19cc0997df34ed99cde17b5879"
server_key_iv: "75ca701b8d45af6ea45cdf406e5cfe63"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.325608 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:13.349901 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:13.350296 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:13.350461 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:13.354995 20044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46775
I20260925 13:16:13.355028 20110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46775 every 8 connection(s)
I20260925 13:16:13.355391 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:13.356383 20111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:13.358886 20111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f: Bootstrap starting.
I20260925 13:16:13.359500 20111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:13.359807 20111 log.cc:824] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:13.360368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20044
I20260925 13:16:13.360495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:13.360605 20111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f: No bootstrap required, opened a new log
I20260925 13:16:13.360716 392 external_mini_cluster.cc:1428] Setting key a2b1d133e623bdf51ec7b3e7cb517253
I20260925 13:16:13.363300 20111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e429f271d9a4604abd42001260c679f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260925 13:16:13.363499 20111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:13.363538 20111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e429f271d9a4604abd42001260c679f, State: Initialized, Role: FOLLOWER
I20260925 13:16:13.363662 20111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e429f271d9a4604abd42001260c679f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260925 13:16:13.363739 20111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:13.363786 20111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:13.363833 20111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.364568 20111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e429f271d9a4604abd42001260c679f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260925 13:16:13.364660 20111 leader_election.cc:302] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e429f271d9a4604abd42001260c679f; no voters:
I20260925 13:16:13.364902 20111 leader_election.cc:288] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:13.364930 20116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:13.365044 20116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [term 1 LEADER]: Becoming Leader. State: Replica: 1e429f271d9a4604abd42001260c679f, State: Running, Role: LEADER
I20260925 13:16:13.365123 20116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e429f271d9a4604abd42001260c679f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260925 13:16:13.365190 20111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:13.365432 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e429f271d9a4604abd42001260c679f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e429f271d9a4604abd42001260c679f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } } }
I20260925 13:16:13.365501 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:13.365747 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e429f271d9a4604abd42001260c679f. Latest consensus state: current_term: 1 leader_uuid: "1e429f271d9a4604abd42001260c679f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e429f271d9a4604abd42001260c679f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } } }
I20260925 13:16:13.365893 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:13.366093 20123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:13.366665 20123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:13.366916 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.361258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.98.62:46775)
0925 13:16:13.361483 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.361491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.361517 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:16:13.362046 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.362053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.362112 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.362226 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:13.362484 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.363368 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.364444 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.364995 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.365119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.365922 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:13.365925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:13.365926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:13.366157 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:13.366203 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:13.366217 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:13.366285 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.366563 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.366661 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260925 13:16:13.368573 20123 catalog_manager.cc:1347] Generated new cluster ID: 5deed0131a024bef8490aa8aafdcda92
I20260925 13:16:13.368633 20123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:13.372699 20123 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:13.373381 20123 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:13.378403 20123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f: Generated new TSK 0
I20260925 13:16:13.378592 20123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:13.382400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46775
--builtin_ntp_servers=127.0.98.20:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:16:13.478995 20135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:13.479221 20135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:13.479245 20135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:13.480471 20135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:13.480513 20135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:13.480545 20135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:16:13.481984 20135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:13.482259 20135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:13.482594 20135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:13.488677 20141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.488845 20143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.488668 20140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:13.489339 20135 server_base.cc:1034] running on GCE node
I20260925 13:16:13.489566 20135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:13.489845 20135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:13.491009 20135 hybrid_clock.cc:648] HybridClock initialized: now 1790342173491005 us; error 39 us; skew 500 ppm
I20260925 13:16:13.492377 20135 webserver.cc:460] Webserver started at http://127.0.98.1:42931/ using document root <none> and password file <none>
I20260925 13:16:13.492643 20135 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:13.492714 20135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:13.492869 20135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:13.494078 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data/instance:
uuid: "223cb665ef27417f9607e7bd35d27291"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "e43e634d841e3b9893aecae240be644b"
server_key_iv: "992678403c27d3b95886931e9c4ee814"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.494447 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal/instance:
uuid: "223cb665ef27417f9607e7bd35d27291"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "e43e634d841e3b9893aecae240be644b"
server_key_iv: "992678403c27d3b95886931e9c4ee814"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.495873 20135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:13.496737 20149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.496968 20135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:13.497093 20135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal
uuid: "223cb665ef27417f9607e7bd35d27291"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "e43e634d841e3b9893aecae240be644b"
server_key_iv: "992678403c27d3b95886931e9c4ee814"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.497177 20135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:13.518630 20135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:13.519006 20135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:13.519171 20135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:13.519428 20135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:13.519783 20135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:13.519840 20135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.519886 20135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:13.519910 20135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.525982 20135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33781
I20260925 13:16:13.526044 20262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33781 every 8 connection(s)
I20260925 13:16:13.526427 20135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:16:13.528544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20135
I20260925 13:16:13.528663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:16:13.528854 392 external_mini_cluster.cc:1428] Setting key ce144967ae3411b2b984e0c86a944e61
I20260925 13:16:13.529920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46775
--builtin_ntp_servers=127.0.98.20:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 13:16:13.532382 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.527291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52931 (local address 127.0.98.62:46775)
0925 13:16:13.527380 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.527383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.527590 (+ 207us) server_negotiation.cc:407] Connection header received
0925 13:16:13.528079 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.528080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.528095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.528141 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:13.528967 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.529734 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.530858 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.531083 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.531163 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.532081 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:13.532083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:13.532083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:13.532089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:13.532105 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:13.532109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:13.532158 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.532255 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.532300 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 13:16:13.532635 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.527252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46775 (local address 127.0.98.1:52931)
0925 13:16:13.527466 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.527501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.527998 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.528224 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.528228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.528236 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:13.528864 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.528869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.529855 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.529857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.530751 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.530754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.531172 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.531173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.531314 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.531920 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:13.531921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:13.531958 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:13.531983 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:13.532157 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:13.532163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:13.532166 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.532257 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.532368 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260925 13:16:13.533133 20263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46775
I20260925 13:16:13.533254 20263 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:13.533447 20263 heartbeater.cc:511] Master 127.0.98.62:46775 requested a full tablet report, sending...
I20260925 13:16:13.534085 20075 ts_manager.cc:194] Registered new tserver with Master: 223cb665ef27417f9607e7bd35d27291 (127.0.98.1:33781)
I20260925 13:16:13.535115 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52931
W20260925 13:16:13.614043 20266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:13.614260 20266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:13.614282 20266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:13.615515 20266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:13.615563 20266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:13.615589 20266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:16:13.616986 20266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:13.617247 20266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:13.617573 20266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:13.622637 20274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.622699 20272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.622646 20271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:13.622938 20266 server_base.cc:1034] running on GCE node
I20260925 13:16:13.623142 20266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:13.623379 20266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:13.624555 20266 hybrid_clock.cc:648] HybridClock initialized: now 1790342173624547 us; error 39 us; skew 500 ppm
I20260925 13:16:13.625953 20266 webserver.cc:460] Webserver started at http://127.0.98.2:39995/ using document root <none> and password file <none>
I20260925 13:16:13.626225 20266 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:13.626299 20266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:13.626454 20266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:13.627548 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data/instance:
uuid: "854a2f3068104b3bbce5adbafce7d7f8"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "76573fcf511478ce3f94126e56e72666"
server_key_iv: "084fbfc602dacdbea5591e26aa44ed61"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.627907 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal/instance:
uuid: "854a2f3068104b3bbce5adbafce7d7f8"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "76573fcf511478ce3f94126e56e72666"
server_key_iv: "084fbfc602dacdbea5591e26aa44ed61"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.629328 20266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:13.630138 20280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.630311 20266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.630403 20266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal
uuid: "854a2f3068104b3bbce5adbafce7d7f8"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "76573fcf511478ce3f94126e56e72666"
server_key_iv: "084fbfc602dacdbea5591e26aa44ed61"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.630492 20266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:13.652817 20266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:13.653170 20266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:13.653337 20266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:13.653599 20266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:13.653970 20266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:13.654026 20266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.654067 20266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:13.654098 20266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.660080 20266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38949
I20260925 13:16:13.660130 20393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38949 every 8 connection(s)
I20260925 13:16:13.660490 20266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:16:13.665597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20266
I20260925 13:16:13.665783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:16:13.666055 392 external_mini_cluster.cc:1428] Setting key 5c7d15e57b3e52e415be38447ccd0c4c
I20260925 13:16:13.666312 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.661248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55507 (local address 127.0.98.62:46775)
0925 13:16:13.661302 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.661304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.661564 (+ 260us) server_negotiation.cc:407] Connection header received
0925 13:16:13.662202 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.662203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.662216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.662271 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:13.663009 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.663519 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.664540 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.664738 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.664817 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.665879 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:13.665881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:13.665882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:13.665888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:13.665908 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:13.665913 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:13.666011 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.666186 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.666216 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:16:13.666550 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.661233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46775 (local address 127.0.98.2:55507)
0925 13:16:13.661427 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.661464 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.662077 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.662351 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.662356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.662372 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:13.662895 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.662899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.663621 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.663622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.664446 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.664448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.664847 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.664848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.664996 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.665720 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:13.665720 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:13.665751 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:13.665796 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:13.666042 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:13.666050 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:13.666053 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.666133 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.666252 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260925 13:16:13.666992 20394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46775
I20260925 13:16:13.667104 20394 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:13.667164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46775
--builtin_ntp_servers=127.0.98.20:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 13:16:13.667320 20394 heartbeater.cc:511] Master 127.0.98.62:46775 requested a full tablet report, sending...
I20260925 13:16:13.667729 20075 ts_manager.cc:194] Registered new tserver with Master: 854a2f3068104b3bbce5adbafce7d7f8 (127.0.98.2:38949)
I20260925 13:16:13.668251 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55507
W20260925 13:16:13.750645 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:13.750922 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:13.750960 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:13.752319 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:13.752391 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:13.752437 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:16:13.753923 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:13.754233 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:13.754567 20397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:13.759766 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.759771 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:13.759786 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:13.760192 20397 server_base.cc:1034] running on GCE node
I20260925 13:16:13.760461 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:13.760679 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:13.761837 20397 hybrid_clock.cc:648] HybridClock initialized: now 1790342173761822 us; error 40 us; skew 500 ppm
I20260925 13:16:13.763232 20397 webserver.cc:460] Webserver started at http://127.0.98.3:41197/ using document root <none> and password file <none>
I20260925 13:16:13.763494 20397 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:13.763568 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:13.763718 20397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:13.764827 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data/instance:
uuid: "7514b43a1a374811803c1ba8f5edba72"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "e5ae290292c48107e4989fe544f48648"
server_key_iv: "635d6652d7e369b73c4a7b110499eb7f"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.765239 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7514b43a1a374811803c1ba8f5edba72"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "e5ae290292c48107e4989fe544f48648"
server_key_iv: "635d6652d7e369b73c4a7b110499eb7f"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.766902 20397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:16:13.767762 20411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.767933 20397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:13.768047 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal
uuid: "7514b43a1a374811803c1ba8f5edba72"
format_stamp: "Formatted at 2026-09-25 13:16:13 on dist-test-slave-2d0s"
server_key: "e5ae290292c48107e4989fe544f48648"
server_key_iv: "635d6652d7e369b73c4a7b110499eb7f"
server_key_version: "encryptionkey@0"
I20260925 13:16:13.768126 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:13.778837 20397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:13.779196 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:13.779362 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:13.779628 20397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:16:13.779973 20397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:16:13.780027 20397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.780071 20397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:16:13.780095 20397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:13.786283 20397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42443
I20260925 13:16:13.786338 20524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42443 every 8 connection(s)
I20260925 13:16:13.786737 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:16:13.792361 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.787591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55715 (local address 127.0.98.62:46775)
0925 13:16:13.787654 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.787658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.787886 (+ 228us) server_negotiation.cc:407] Connection header received
0925 13:16:13.788402 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.788404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.788415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.788483 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:13.789199 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.789795 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.790837 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.791043 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.791104 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.792091 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:13.792093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:13.792093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:13.792097 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:13.792112 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:13.792122 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:13.792177 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.792266 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.792290 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:16:13.792601 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.787549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46775 (local address 127.0.98.3:55715)
0925 13:16:13.787747 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.787791 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.788304 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.788549 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.788553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.788562 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:13.789088 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.789093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.789894 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.789895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.790743 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.790745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.791129 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.791130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.791247 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.791917 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:16:13.791917 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:13.791948 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:16:13.791973 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:13.792193 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:13.792198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:13.792201 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.792256 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.792349 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260925 13:16:13.792635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20397
I20260925 13:16:13.792765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:16:13.793005 392 external_mini_cluster.cc:1428] Setting key cf840328b8eeab2dceb2b5cf6edeac62
I20260925 13:16:13.793313 20525 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46775
I20260925 13:16:13.793463 20525 heartbeater.cc:464] Registering TS with master...
I20260925 13:16:13.793757 20525 heartbeater.cc:511] Master 127.0.98.62:46775 requested a full tablet report, sending...
I20260925 13:16:13.794384 20075 ts_manager.cc:194] Registered new tserver with Master: 7514b43a1a374811803c1ba8f5edba72 (127.0.98.3:42443)
I20260925 13:16:13.794799 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55715
I20260925 13:16:13.805280 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:16:13.809917 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.806845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.0.98.62:46775)
0925 13:16:13.806920 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.806922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.807019 (+ 97us) server_negotiation.cc:407] Connection header received
0925 13:16:13.807067 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.807068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.807079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.807118 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:13.807260 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.807751 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.808484 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.808692 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.808757 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.809489 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:13.809490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:13.809491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:13.809495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:13.809512 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:13.809516 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:13.809568 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.809764 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.809789 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:16:13.811764 20075 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:16:13.813418 20075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:16:13.820560 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.815919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33781 (local address 127.0.0.1:34488)
0925 13:16:13.816469 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.816516 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.816617 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.816920 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.816923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.816928 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.817043 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.817046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.818431 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.818432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.819367 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.819370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.820228 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.820229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.820312 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.820327 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.820402 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.820465 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":53,"thread_start_us":117,"threads_started":1}
I20260925 13:16:13.820578 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.816092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.0.98.1:33781)
0925 13:16:13.816305 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.816312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.816703 (+ 391us) server_negotiation.cc:407] Connection header received
0925 13:16:13.816752 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.816754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.816771 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.816844 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.817127 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.818340 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.819469 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.820105 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.820191 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.820287 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.820435 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.820487 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260925 13:16:13.820636 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.816157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.0.98.3:42443)
0925 13:16:13.816335 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.816339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.816602 (+ 263us) server_negotiation.cc:407] Connection header received
0925 13:16:13.816786 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.816789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.816807 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.816863 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.817125 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.817991 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.819505 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.820128 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.820191 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.820287 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.820504 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.820544 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260925 13:16:13.820865 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.815910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42443 (local address 127.0.0.1:40300)
0925 13:16:13.816512 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.816521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.816617 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.816958 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.816959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.816961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.817039 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.817040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.818075 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.818076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.819390 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.819392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.820347 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.820348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.820407 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.820410 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.820742 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.820805 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":55,"thread_start_us":449,"threads_started":1}
I20260925 13:16:13.821182 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.816292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.0.98.2:38949)
0925 13:16:13.816463 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.816466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.816702 (+ 236us) server_negotiation.cc:407] Connection header received
0925 13:16:13.816752 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.816754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.816771 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.816843 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.817126 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.818321 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.819845 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.820571 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.820880 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.821009 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.821037 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.821082 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260925 13:16:13.821300 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.815910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38949 (local address 127.0.0.1:47632)
0925 13:16:13.816470 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.816517 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.816617 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.816920 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.816923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.816926 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.817042 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.817045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.818417 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.818417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.819746 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.819748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.820717 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.820719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.820784 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.820788 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.820830 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.820864 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":32,"thread_start_us":133,"threads_started":1}
I20260925 13:16:13.821979 20459 tablet_service.cc:1467] Processing CreateTablet for tablet 44c05385ae9c4c10975a5f677babdb21 (DEFAULT_TABLE table=test-table [id=ccfd22eb63cb41af883e359571a960ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:13.822046 20328 tablet_service.cc:1467] Processing CreateTablet for tablet 44c05385ae9c4c10975a5f677babdb21 (DEFAULT_TABLE table=test-table [id=ccfd22eb63cb41af883e359571a960ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:13.822072 20197 tablet_service.cc:1467] Processing CreateTablet for tablet 44c05385ae9c4c10975a5f677babdb21 (DEFAULT_TABLE table=test-table [id=ccfd22eb63cb41af883e359571a960ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:16:13.822389 20328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c05385ae9c4c10975a5f677babdb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:13.822392 20459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c05385ae9c4c10975a5f677babdb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:13.822388 20197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c05385ae9c4c10975a5f677babdb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:13.825245 20539 tablet_bootstrap.cc:492] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Bootstrap starting.
I20260925 13:16:13.825961 20540 tablet_bootstrap.cc:492] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Bootstrap starting.
I20260925 13:16:13.826107 20539 tablet_bootstrap.cc:654] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:13.826119 20541 tablet_bootstrap.cc:492] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291: Bootstrap starting.
I20260925 13:16:13.826475 20539 log.cc:824] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:13.826658 20540 tablet_bootstrap.cc:654] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:13.826828 20541 tablet_bootstrap.cc:654] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:13.826990 20540 log.cc:824] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:13.827186 20541 log.cc:824] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:13.827689 20539 tablet_bootstrap.cc:492] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: No bootstrap required, opened a new log
I20260925 13:16:13.827764 20540 tablet_bootstrap.cc:492] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: No bootstrap required, opened a new log
I20260925 13:16:13.827770 20539 ts_tablet_manager.cc:1397] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 13:16:13.827836 20540 ts_tablet_manager.cc:1397] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:13.827901 20541 tablet_bootstrap.cc:492] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291: No bootstrap required, opened a new log
I20260925 13:16:13.827967 20541 ts_tablet_manager.cc:1397] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:16:13.830067 20540 raft_consensus.cc:348] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.830265 20540 raft_consensus.cc:374] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:13.830307 20540 raft_consensus.cc:729] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7514b43a1a374811803c1ba8f5edba72, State: Initialized, Role: FOLLOWER
I20260925 13:16:13.830428 20540 consensus_queue.cc:260] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.830746 20540 ts_tablet_manager.cc:1428] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:16:13.830651 20539 raft_consensus.cc:348] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.830660 20541 raft_consensus.cc:348] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.830758 20525 heartbeater.cc:503] Master 127.0.98.62:46775 was elected leader, sending a full tablet report...
I20260925 13:16:13.830807 20541 raft_consensus.cc:374] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:13.830807 20539 raft_consensus.cc:374] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:13.830843 20541 raft_consensus.cc:729] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223cb665ef27417f9607e7bd35d27291, State: Initialized, Role: FOLLOWER
I20260925 13:16:13.830843 20539 raft_consensus.cc:729] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854a2f3068104b3bbce5adbafce7d7f8, State: Initialized, Role: FOLLOWER
I20260925 13:16:13.830974 20541 consensus_queue.cc:260] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.831010 20539 consensus_queue.cc:260] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.831249 20541 ts_tablet_manager.cc:1428] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 13:16:13.831276 20394 heartbeater.cc:503] Master 127.0.98.62:46775 was elected leader, sending a full tablet report...
I20260925 13:16:13.831367 20263 heartbeater.cc:503] Master 127.0.98.62:46775 was elected leader, sending a full tablet report...
I20260925 13:16:13.831480 20539 ts_tablet_manager.cc:1428] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260925 13:16:13.911056 20395 tablet.cc:2369] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:16:13.932096 20545 raft_consensus.cc:482] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:16:13.932277 20545 raft_consensus.cc:504] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.932796 20545 leader_election.cc:288] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223cb665ef27417f9607e7bd35d27291 (127.0.98.1:33781), 854a2f3068104b3bbce5adbafce7d7f8 (127.0.98.2:38949)
I20260925 13:16:13.935819 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.932946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33781 (local address 127.0.98.3:56679)
0925 13:16:13.932999 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.933012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.933056 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.933240 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.933241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.933243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.933322 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.933324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.934151 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.934152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.934980 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.934982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.935537 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.935539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.935621 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.935624 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.935666 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.935730 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260925 13:16:13.935820 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.933036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56679 (local address 127.0.98.1:33781)
0925 13:16:13.933095 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.933097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.933105 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:16:13.933134 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.933135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.933148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.933213 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.933430 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.934071 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.935085 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.935455 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.935513 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.935546 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.935695 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.935743 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:16:13.935851 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.933057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33305 (local address 127.0.98.2:38949)
0925 13:16:13.933102 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.933104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.933241 (+ 137us) server_negotiation.cc:407] Connection header received
0925 13:16:13.933276 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.933277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.933291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.933363 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:16:13.933588 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.934171 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.935106 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.935463 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.935550 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.935559 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.935738 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.935779 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:13.935854 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.932986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38949 (local address 127.0.98.3:33305)
0925 13:16:13.933173 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.933182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:13.933230 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:13.933405 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:13.933406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:13.933408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:16:13.933511 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.933513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.934254 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.934255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.934994 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:13.934996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.935568 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.935569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:13.935642 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.935645 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:16:13.935724 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:16:13.935760 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
I20260925 13:16:13.936242 20217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c05385ae9c4c10975a5f677babdb21" candidate_uuid: "7514b43a1a374811803c1ba8f5edba72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223cb665ef27417f9607e7bd35d27291" is_pre_election: true
I20260925 13:16:13.936343 20348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c05385ae9c4c10975a5f677babdb21" candidate_uuid: "7514b43a1a374811803c1ba8f5edba72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854a2f3068104b3bbce5adbafce7d7f8" is_pre_election: true
I20260925 13:16:13.936426 20217 raft_consensus.cc:2454] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7514b43a1a374811803c1ba8f5edba72 in term 0.
I20260925 13:16:13.936529 20348 raft_consensus.cc:2454] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7514b43a1a374811803c1ba8f5edba72 in term 0.
I20260925 13:16:13.936667 20415 leader_election.cc:302] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223cb665ef27417f9607e7bd35d27291, 7514b43a1a374811803c1ba8f5edba72; no voters:
I20260925 13:16:13.936785 20545 raft_consensus.cc:2781] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:16:13.936837 20545 raft_consensus.cc:482] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:16:13.936868 20545 raft_consensus.cc:3037] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.937587 20545 raft_consensus.cc:504] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.937774 20545 leader_election.cc:288] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [CANDIDATE]: Term 1 election: Requested vote from peers 223cb665ef27417f9607e7bd35d27291 (127.0.98.1:33781), 854a2f3068104b3bbce5adbafce7d7f8 (127.0.98.2:38949)
I20260925 13:16:13.937937 20348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c05385ae9c4c10975a5f677babdb21" candidate_uuid: "7514b43a1a374811803c1ba8f5edba72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854a2f3068104b3bbce5adbafce7d7f8"
I20260925 13:16:13.937937 20217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c05385ae9c4c10975a5f677babdb21" candidate_uuid: "7514b43a1a374811803c1ba8f5edba72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223cb665ef27417f9607e7bd35d27291"
I20260925 13:16:13.938019 20348 raft_consensus.cc:3037] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.938019 20217 raft_consensus.cc:3037] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:13.938954 20217 raft_consensus.cc:2454] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7514b43a1a374811803c1ba8f5edba72 in term 1.
I20260925 13:16:13.938954 20348 raft_consensus.cc:2454] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7514b43a1a374811803c1ba8f5edba72 in term 1.
I20260925 13:16:13.939200 20414 leader_election.cc:302] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7514b43a1a374811803c1ba8f5edba72, 854a2f3068104b3bbce5adbafce7d7f8; no voters:
I20260925 13:16:13.939311 20545 raft_consensus.cc:2781] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:13.939494 20545 raft_consensus.cc:686] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 1 LEADER]: Becoming Leader. State: Replica: 7514b43a1a374811803c1ba8f5edba72, State: Running, Role: LEADER
I20260925 13:16:13.939621 20545 consensus_queue.cc:237] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } }
I20260925 13:16:13.940614 20073 catalog_manager.cc:5495] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7514b43a1a374811803c1ba8f5edba72 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7514b43a1a374811803c1ba8f5edba72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7514b43a1a374811803c1ba8f5edba72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 } health_report { overall_health: UNKNOWN } } }
I20260925 13:16:13.966626 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.964010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.0.98.3:42443)
0925 13:16:13.964062 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:13.964065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:13.964072 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:16:13.964113 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:13.964114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:13.964139 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:13.964209 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:13.964312 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.964845 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.965733 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:13.965949 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:13.966012 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:13.966206 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:13.966319 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:13.966389 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:13.966516 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 13:16:13.966538 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:16:13.968775 20348 raft_consensus.cc:1261] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 1 FOLLOWER]: Refusing update from remote peer 7514b43a1a374811803c1ba8f5edba72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:13.968991 20217 raft_consensus.cc:1261] T 44c05385ae9c4c10975a5f677babdb21 P 223cb665ef27417f9607e7bd35d27291 [term 1 FOLLOWER]: Refusing update from remote peer 7514b43a1a374811803c1ba8f5edba72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:13.969199 20545 consensus_queue.cc:1035] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854a2f3068104b3bbce5adbafce7d7f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:13.969358 20549 consensus_queue.cc:1035] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223cb665ef27417f9607e7bd35d27291" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:13.972342 20552 mvcc.cc:205] Tried to move back new op lower bound from 7333241544572862464 to 7333241544457617408. Current Snapshot: MvccSnapshot[applied={T|T < 7333241544572862464}]
I20260925 13:16:13.972661 20554 mvcc.cc:205] Tried to move back new op lower bound from 7333241544572862464 to 7333241544457617408. Current Snapshot: MvccSnapshot[applied={T|T < 7333241544572862464}]
I20260925 13:16:13.973165 20553 mvcc.cc:205] Tried to move back new op lower bound from 7333241544572862464 to 7333241544457617408. Current Snapshot: MvccSnapshot[applied={T|T < 7333241544572862464}]
I20260925 13:16:13.975740 20074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:13.975880 20074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:16:13.976794 20074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e429f271d9a4604abd42001260c679f: Sending DeleteTablet for 3 replicas of tablet 44c05385ae9c4c10975a5f677babdb21
I20260925 13:16:13.977223 20197 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c05385ae9c4c10975a5f677babdb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:13 UTC) from {username='slave'} at 127.0.0.1:34488
I20260925 13:16:13.977217 20328 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c05385ae9c4c10975a5f677babdb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:13 UTC) from {username='slave'} at 127.0.0.1:47632
I20260925 13:16:13.977581 20459 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c05385ae9c4c10975a5f677babdb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:16:13 UTC) from {username='slave'} at 127.0.0.1:40300
I20260925 13:16:13.977849 20562 tablet_replica.cc:330] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: stopping tablet replica
I20260925 13:16:13.977887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20135
I20260925 13:16:13.977984 20562 raft_consensus.cc:2229] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:16:13.978091 20562 raft_consensus.cc:2258] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:13.978298 20564 tablet_replica.cc:330] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: stopping tablet replica
I20260925 13:16:13.978379 20564 raft_consensus.cc:2229] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:16:13.978507 20564 raft_consensus.cc:2258] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:16:13.978785 20562 ts_tablet_manager.cc:1905] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:13.979255 20564 ts_tablet_manager.cc:1905] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:16:13.980592 20562 ts_tablet_manager.cc:1918] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:13.980679 20562 log.cc:1196] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-1/wal/wals/44c05385ae9c4c10975a5f677babdb21
I20260925 13:16:13.980923 20564 ts_tablet_manager.cc:1918] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:16:13.980955 20562 ts_tablet_manager.cc:1939] T 44c05385ae9c4c10975a5f677babdb21 P 854a2f3068104b3bbce5adbafce7d7f8: Deleting consensus metadata
I20260925 13:16:13.981002 20564 log.cc:1196] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Deleting WAL directory at /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790341932215442-392-0/minicluster-data/ts-2/wal/wals/44c05385ae9c4c10975a5f677babdb21
I20260925 13:16:13.981256 20564 ts_tablet_manager.cc:1939] T 44c05385ae9c4c10975a5f677babdb21 P 7514b43a1a374811803c1ba8f5edba72: Deleting consensus metadata
I20260925 13:16:13.981472 20061 catalog_manager.cc:4841] TS 854a2f3068104b3bbce5adbafce7d7f8 (127.0.98.2:38949): tablet 44c05385ae9c4c10975a5f677babdb21 (table test-table [id=ccfd22eb63cb41af883e359571a960ce]) successfully deleted
I20260925 13:16:13.981644 20059 catalog_manager.cc:4841] TS 7514b43a1a374811803c1ba8f5edba72 (127.0.98.3:42443): tablet 44c05385ae9c4c10975a5f677babdb21 (table test-table [id=ccfd22eb63cb41af883e359571a960ce]) successfully deleted
W20260925 13:16:13.984861 20062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33781 (error 108)
I20260925 13:16:13.985481 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:13.985319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33781 (local address 127.0.0.1:34490)
0925 13:16:13.985388 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:13.985437 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 13:16:13.985512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20266
W20260925 13:16:13.989277 20062 catalog_manager.cc:4584] TS 223cb665ef27417f9607e7bd35d27291 (127.0.98.1:33781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44c05385ae9c4c10975a5f677babdb21: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33781: connect: Connection refused (error 111)
I20260925 13:16:13.994210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20397
I20260925 13:16:13.998945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20044
2026-09-25T13:16:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (801 ms)
[----------] 6 tests from EncryptionPolicyTest (8745 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:14 dist-test-slave-2d0s krb5kdc[20568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:14 dist-test-slave-2d0s krb5kdc[20568](info): set up 2 sockets
Sep 25 13:16:14 dist-test-slave-2d0s krb5kdc[20568](info): commencing operation
krb5kdc: starting...
W20260925 13:16:15.049885 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:15 dist-test-slave-2d0s krb5kdc[20568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:15.168578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:15.250723 20581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:15.250994 20581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:15.251035 20581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:15.252199 20581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:15.252274 20581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:15.252297 20581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:15.252345 20581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:15.252368 20581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:15.253974 20581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:15.254261 20581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:15.254509 20581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:15.259359 20587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:15.259373 20586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:15.259373 20589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:15.259862 20581 server_base.cc:1034] running on GCE node
I20260925 13:16:15.260126 20581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:15.260432 20581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:15.261577 20581 hybrid_clock.cc:648] HybridClock initialized: now 1790342175261554 us; error 35 us; skew 500 ppm
Sep 25 13:16:15 dist-test-slave-2d0s krb5kdc[20568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:15.271270 20581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:15.271836 20581 webserver.cc:460] Webserver started at http://127.0.0.1:35617/ using document root <none> and password file <none>
I20260925 13:16:15.272096 20581 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:15.272166 20581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:15.272323 20581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:15.273506 20581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "c388ee39b1de40ba96b4c5c17fd87e0f"
format_stamp: "Formatted at 2026-09-25 13:16:15 on dist-test-slave-2d0s"
server_key: "3b92d72e559bacad1549b2e5a9b92d6f"
server_key_iv: "fb1b0b96c04cd1054f0486ba207d3cc0"
server_key_version: "encryptionkey@0"
I20260925 13:16:15.273907 20581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "c388ee39b1de40ba96b4c5c17fd87e0f"
format_stamp: "Formatted at 2026-09-25 13:16:15 on dist-test-slave-2d0s"
server_key: "3b92d72e559bacad1549b2e5a9b92d6f"
server_key_iv: "fb1b0b96c04cd1054f0486ba207d3cc0"
server_key_version: "encryptionkey@0"
I20260925 13:16:15.275267 20581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:15.276021 20596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:15.276192 20581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:16:15.276301 20581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "c388ee39b1de40ba96b4c5c17fd87e0f"
format_stamp: "Formatted at 2026-09-25 13:16:15 on dist-test-slave-2d0s"
server_key: "3b92d72e559bacad1549b2e5a9b92d6f"
server_key_iv: "fb1b0b96c04cd1054f0486ba207d3cc0"
server_key_version: "encryptionkey@0"
I20260925 13:16:15.276396 20581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:15.287267 20581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:15.289196 20581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:15.289374 20581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:15.293722 20581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39689
I20260925 13:16:15.293776 20648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39689 every 8 connection(s)
I20260925 13:16:15.294142 20581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:15.294999 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:15.297291 20649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f: Bootstrap starting.
I20260925 13:16:15.297844 20649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:15.298074 20649 log.cc:824] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:15.298714 20649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f: No bootstrap required, opened a new log
I20260925 13:16:15.300251 20649 raft_consensus.cc:348] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260925 13:16:15.300395 20649 raft_consensus.cc:374] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:15.300418 20649 raft_consensus.cc:729] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c388ee39b1de40ba96b4c5c17fd87e0f, State: Initialized, Role: FOLLOWER
I20260925 13:16:15.300568 20649 consensus_queue.cc:260] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260925 13:16:15.300652 20649 raft_consensus.cc:388] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:15.300681 20649 raft_consensus.cc:482] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:15.300715 20649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:15.301198 20649 raft_consensus.cc:504] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260925 13:16:15.301288 20649 leader_election.cc:302] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c388ee39b1de40ba96b4c5c17fd87e0f; no voters:
I20260925 13:16:15.301412 20649 leader_election.cc:288] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:15.301478 20652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:15.301630 20652 raft_consensus.cc:686] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [term 1 LEADER]: Becoming Leader. State: Replica: c388ee39b1de40ba96b4c5c17fd87e0f, State: Running, Role: LEADER
I20260925 13:16:15.301712 20649 sys_catalog.cc:564] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:15.301735 20652 consensus_queue.cc:237] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260925 13:16:15.302064 20654 sys_catalog.cc:455] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c388ee39b1de40ba96b4c5c17fd87e0f. Latest consensus state: current_term: 1 leader_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } } }
I20260925 13:16:15.302147 20654 sys_catalog.cc:458] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:15.302297 20653 sys_catalog.cc:455] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388ee39b1de40ba96b4c5c17fd87e0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } } }
I20260925 13:16:15.302405 20653 sys_catalog.cc:458] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:15.302835 20659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:15.303328 20659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:15.303846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20581
I20260925 13:16:15.303987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:15.304210 392 external_mini_cluster.cc:1428] Setting key 11b8fd047fb186873f6398cf83930745
I20260925 13:16:15.305146 20659 catalog_manager.cc:1347] Generated new cluster ID: 80dd7c67a8f34fa0a038feb46481133d
I20260925 13:16:15.305207 20659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:16:15 dist-test-slave-2d0s krb5kdc[20568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:16:15.315055 20659 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:15.315757 20659 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:15.316114 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:15.304688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56971 (local address 127.0.0.1:39689)
0925 13:16:15.305407 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:15.305414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:15.305437 (+ 23us) server_negotiation.cc:407] Connection header received
0925 13:16:15.305525 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:15.305530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:15.305570 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:15.305727 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:15.306369 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:15.307289 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:15.308060 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:15.308512 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:15.308634 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:15.311536 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:15.311542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:15.311542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:15.311557 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:15.314290 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:15.314775 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:15.314779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:15.314779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:15.314897 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:15.315219 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:15.315220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:15.315221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:15.315446 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:15.315645 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:15.315735 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 13:16:15.315922 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":72,"threads_started":1}
I20260925 13:16:15.317097 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:15.325014 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:15.318675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55731 (local address 127.0.0.1:39689)
0925 13:16:15.318714 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:15.318716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:15.318841 (+ 125us) server_negotiation.cc:407] Connection header received
0925 13:16:15.318899 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:15.318900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:15.318909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:15.318949 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:15.319530 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:15.320064 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:15.320765 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:15.320996 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:15.321087 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:15.322386 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:15.322388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:15.322388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:15.322395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:15.323800 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:15.324174 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:15.324175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:15.324176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:15.324229 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:15.324465 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:15.324466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:15.324467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:15.324593 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:15.324714 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:15.324830 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 13:16:15.324893 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:16:15.326464 20659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c388ee39b1de40ba96b4c5c17fd87e0f: Generated new TSK 0
I20260925 13:16:15.326670 20659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:15.333024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20581
W20260925 13:16:15.337476 20566 connection.cc:535] client connection to 127.0.0.1:39689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39689): Connection reset by peer (error 104)
2026-09-25T13:16:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:15 dist-test-slave-2d0s krb5kdc[20680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:15 dist-test-slave-2d0s krb5kdc[20680](info): set up 2 sockets
Sep 25 13:16:15 dist-test-slave-2d0s krb5kdc[20680](info): commencing operation
krb5kdc: starting...
W20260925 13:16:16.353919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:16 dist-test-slave-2d0s krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:16.469760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:16.552739 20693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:16.552950 20693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:16.552973 20693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:16.554136 20693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:16.554176 20693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:16.554190 20693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:16.554203 20693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:16.554260 20693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:16.555680 20693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:16.555963 20693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:16.556200 20693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:16.561151 20701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:16.561156 20698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:16.561156 20699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:16.561483 20693 server_base.cc:1034] running on GCE node
I20260925 13:16:16.561776 20693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:16.562077 20693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:16.563251 20693 hybrid_clock.cc:648] HybridClock initialized: now 1790342176563235 us; error 48 us; skew 500 ppm
Sep 25 13:16:16 dist-test-slave-2d0s krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:16.565934 20693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:16.566450 20693 webserver.cc:460] Webserver started at http://127.0.0.1:41327/ using document root <none> and password file <none>
I20260925 13:16:16.566684 20693 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:16.566753 20693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:16.566916 20693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:16.568048 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "8be72690c7af4aa6a4537601b9fd9de2"
format_stamp: "Formatted at 2026-09-25 13:16:16 on dist-test-slave-2d0s"
server_key: "0edad96cd4158c5cc704b616ac10b058"
server_key_iv: "9f2fe4a5357bb549c1b7a829a1a7c922"
server_key_version: "encryptionkey@0"
I20260925 13:16:16.568426 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "8be72690c7af4aa6a4537601b9fd9de2"
format_stamp: "Formatted at 2026-09-25 13:16:16 on dist-test-slave-2d0s"
server_key: "0edad96cd4158c5cc704b616ac10b058"
server_key_iv: "9f2fe4a5357bb549c1b7a829a1a7c922"
server_key_version: "encryptionkey@0"
I20260925 13:16:16.569878 20693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 13:16:16.570628 20708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:16.570815 20693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:16.570917 20693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "8be72690c7af4aa6a4537601b9fd9de2"
format_stamp: "Formatted at 2026-09-25 13:16:16 on dist-test-slave-2d0s"
server_key: "0edad96cd4158c5cc704b616ac10b058"
server_key_iv: "9f2fe4a5357bb549c1b7a829a1a7c922"
server_key_version: "encryptionkey@0"
I20260925 13:16:16.571000 20693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:16.585590 20693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:16.588101 20693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:16.588284 20693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:16.592463 20693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43153
I20260925 13:16:16.592497 20760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43153 every 8 connection(s)
I20260925 13:16:16.592864 20693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:16.593832 20761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:16.595153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20693
I20260925 13:16:16.595302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:16.595506 392 external_mini_cluster.cc:1428] Setting key 24f0f346fe3fa676ed2e9c3c863a9a72
I20260925 13:16:16.596728 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2: Bootstrap starting.
I20260925 13:16:16.597470 20761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:16.597802 20761 log.cc:824] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:16.598537 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2: No bootstrap required, opened a new log
I20260925 13:16:16.600728 20761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be72690c7af4aa6a4537601b9fd9de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43153 } }
I20260925 13:16:16.600863 20761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:16.600911 20761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be72690c7af4aa6a4537601b9fd9de2, State: Initialized, Role: FOLLOWER
I20260925 13:16:16.601042 20761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be72690c7af4aa6a4537601b9fd9de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43153 } }
I20260925 13:16:16.601122 20761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:16.601152 20761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:16.601203 20761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:16.601855 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:16.596075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49373 (local address 127.0.0.1:43153)
0925 13:16:16.596278 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:16.596290 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:16.596328 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:16:16.596880 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:16.596890 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:16.596952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:16.597118 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:16.597332 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:16.598746 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:16.599534 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:16.600205 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:16.600347 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:16.601102 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:16.601106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:16.601106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:16.601323 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:16.601366 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:16.601381 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:16.601463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:16.601566 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 13:16:16.601594 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260925 13:16:16.601976 20761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be72690c7af4aa6a4537601b9fd9de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43153 } }
I20260925 13:16:16.602103 20761 leader_election.cc:302] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8be72690c7af4aa6a4537601b9fd9de2; no voters:
I20260925 13:16:16.602316 20761 leader_election.cc:288] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:16.602394 20766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:16.602559 20761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:16.602586 20766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [term 1 LEADER]: Becoming Leader. State: Replica: 8be72690c7af4aa6a4537601b9fd9de2, State: Running, Role: LEADER
I20260925 13:16:16.602766 20766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be72690c7af4aa6a4537601b9fd9de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43153 } }
I20260925 13:16:16.602902 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:16.603924 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8be72690c7af4aa6a4537601b9fd9de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be72690c7af4aa6a4537601b9fd9de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43153 } } }
I20260925 13:16:16.604314 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [sys.catalog]: This master's current role is: LEADER
W20260925 13:16:16.604418 20781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:16.604470 20781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:16.604578 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8be72690c7af4aa6a4537601b9fd9de2. Latest consensus state: current_term: 1 leader_uuid: "8be72690c7af4aa6a4537601b9fd9de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be72690c7af4aa6a4537601b9fd9de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43153 } } }
I20260925 13:16:16.604648 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:16.604877 20787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:16.605067 20787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:16.607182 20787 catalog_manager.cc:1347] Generated new cluster ID: 50d3b0cbd16b40f1838485c26bac55c6
I20260925 13:16:16.607249 20787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:16.608968 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:16.604541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49299 (local address 127.0.0.1:43153)
0925 13:16:16.604582 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:16.604584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:16.605720 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 13:16:16.605789 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:16.605790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:16.605801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:16.605892 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:16.606027 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:16.606761 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:16.607647 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:16.607913 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:16.608007 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:16.608707 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:16.608708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:16.608709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:16.608713 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:16.608729 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:16.608741 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:16.608817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:16.608882 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 13:16:16.608893 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:16:16.623210 20787 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:16.623804 20787 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:16.636109 20787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8be72690c7af4aa6a4537601b9fd9de2: Generated new TSK 0
I20260925 13:16:16.636389 20787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:16.646199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20693
W20260925 13:16:16.650804 20678 connection.cc:535] client connection to 127.0.0.1:43153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43153): Connection reset by peer (error 104)
2026-09-25T13:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:16 dist-test-slave-2d0s krb5kdc[20792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:16 dist-test-slave-2d0s krb5kdc[20792](info): set up 2 sockets
Sep 25 13:16:16 dist-test-slave-2d0s krb5kdc[20792](info): commencing operation
krb5kdc: starting...
W20260925 13:16:17.669822 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:17 dist-test-slave-2d0s krb5kdc[20792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:17.790109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:17.873360 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:17.873598 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:17.873632 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:17.874914 20805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:17.874964 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:17.874979 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:17.874992 20805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:17.875005 20805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:17.876626 20805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:17.876919 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:17.877167 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:17.882232 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:17.882267 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:17.882460 20805 server_base.cc:1034] running on GCE node
W20260925 13:16:17.882257 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:17.882743 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:17.883006 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:17.884191 20805 hybrid_clock.cc:648] HybridClock initialized: now 1790342177884167 us; error 38 us; skew 500 ppm
Sep 25 13:16:17 dist-test-slave-2d0s krb5kdc[20792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:17.886978 20805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:17.887450 20805 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260925 13:16:17.887706 20805 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:17.887778 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:17.887926 20805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:17.889042 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "6e955cbfd74e458a90de590ccb9bab1a"
format_stamp: "Formatted at 2026-09-25 13:16:17 on dist-test-slave-2d0s"
server_key: "122c199b7963686309d8de809c36f382"
server_key_iv: "521a524c90f743d4b8eb3660a4a62e04"
server_key_version: "encryptionkey@0"
I20260925 13:16:17.889429 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e955cbfd74e458a90de590ccb9bab1a"
format_stamp: "Formatted at 2026-09-25 13:16:17 on dist-test-slave-2d0s"
server_key: "122c199b7963686309d8de809c36f382"
server_key_iv: "521a524c90f743d4b8eb3660a4a62e04"
server_key_version: "encryptionkey@0"
I20260925 13:16:17.890915 20805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:17.891707 20820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:17.891925 20805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:17.892036 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "6e955cbfd74e458a90de590ccb9bab1a"
format_stamp: "Formatted at 2026-09-25 13:16:17 on dist-test-slave-2d0s"
server_key: "122c199b7963686309d8de809c36f382"
server_key_iv: "521a524c90f743d4b8eb3660a4a62e04"
server_key_version: "encryptionkey@0"
I20260925 13:16:17.892124 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:17.895720 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:17.897789 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:17.897949 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:17.901932 20805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44271
I20260925 13:16:17.901964 20872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44271 every 8 connection(s)
I20260925 13:16:17.902310 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:17.903334 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:17.905005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20805
I20260925 13:16:17.905184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:17.905387 392 external_mini_cluster.cc:1428] Setting key 380633b15349424923f2f4aab61cd9a8
I20260925 13:16:17.906040 20873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a: Bootstrap starting.
I20260925 13:16:17.906674 20873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:17.907047 20873 log.cc:824] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:17.908197 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:17.906020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34547 (local address 127.0.0.1:44271)
0925 13:16:17.906252 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:17.906260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:17.906286 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:16:17.906783 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:17.906793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:17.906828 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:17.906962 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:17.907720 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:17.907724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:17.907725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:17.907729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:17.907773 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:17.907790 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:17.907888 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:17.907922 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:16:17.907991 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260925 13:16:17.908262 20873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a: No bootstrap required, opened a new log
I20260925 13:16:17.909125 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:17.910386 20873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e955cbfd74e458a90de590ccb9bab1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 13:16:17.910624 20873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:17.910724 20873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e955cbfd74e458a90de590ccb9bab1a, State: Initialized, Role: FOLLOWER
I20260925 13:16:17.910907 20873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e955cbfd74e458a90de590ccb9bab1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 13:16:17.911022 20873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:17.911062 20873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:17.911121 20873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:17.911701 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:17.910677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53783 (local address 127.0.0.1:44271)
0925 13:16:17.910717 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:17.910720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:17.910757 (+ 37us) server_negotiation.cc:407] Connection header received
0925 13:16:17.910845 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:17.910846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:17.910851 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:17.910919 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:17.911475 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:17.911477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:17.911477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:17.911480 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:17.911493 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:17.911497 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:17.911541 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:17.911607 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 13:16:17.911618 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:16:17.911957 20873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e955cbfd74e458a90de590ccb9bab1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 13:16:17.912062 20873 leader_election.cc:302] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e955cbfd74e458a90de590ccb9bab1a; no voters:
I20260925 13:16:17.912241 20873 leader_election.cc:288] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:17.912295 20883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:17.912483 20883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [term 1 LEADER]: Becoming Leader. State: Replica: 6e955cbfd74e458a90de590ccb9bab1a, State: Running, Role: LEADER
I20260925 13:16:17.912531 20873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:17.912598 20883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e955cbfd74e458a90de590ccb9bab1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 13:16:17.912983 20884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e955cbfd74e458a90de590ccb9bab1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e955cbfd74e458a90de590ccb9bab1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } } }
I20260925 13:16:17.913028 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e955cbfd74e458a90de590ccb9bab1a. Latest consensus state: current_term: 1 leader_uuid: "6e955cbfd74e458a90de590ccb9bab1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e955cbfd74e458a90de590ccb9bab1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } } }
I20260925 13:16:17.913105 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:17.913090 20884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:17.913381 20892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:17.913914 20892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:17.915570 20892 catalog_manager.cc:1347] Generated new cluster ID: a353c42a76004921b6254200af70cbc5
I20260925 13:16:17.915632 20892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:17.930923 20892 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:17.931212 20892 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:17.953147 20892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e955cbfd74e458a90de590ccb9bab1a: Generated new TSK 0
I20260925 13:16:17.953400 20892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:17.958571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20805
2026-09-25T13:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:18 dist-test-slave-2d0s krb5kdc[20904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:18 dist-test-slave-2d0s krb5kdc[20904](info): set up 2 sockets
Sep 25 13:16:18 dist-test-slave-2d0s krb5kdc[20904](info): commencing operation
krb5kdc: starting...
W20260925 13:16:18.997871 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:19 dist-test-slave-2d0s krb5kdc[20904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:19.120754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:19.207273 20917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:19.207564 20917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:19.207607 20917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:19.208834 20917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:19.208916 20917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:19.208940 20917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:19.208961 20917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:19.208992 20917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:19.210592 20917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:19.210893 20917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:19.211140 20917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:19.216333 20925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:19.216382 20917 server_base.cc:1034] running on GCE node
W20260925 13:16:19.216355 20922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:19.216346 20923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:19.216863 20917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:19.217175 20917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:19.218361 20917 hybrid_clock.cc:648] HybridClock initialized: now 1790342179218353 us; error 40 us; skew 500 ppm
Sep 25 13:16:19 dist-test-slave-2d0s krb5kdc[20904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:19.220916 20917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:19.221339 20917 webserver.cc:460] Webserver started at http://127.0.0.1:46687/ using document root <none> and password file <none>
I20260925 13:16:19.221526 20917 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:19.221571 20917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:19.221724 20917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:19.222792 20917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "d8107809c76d48e78cfb934a90be7a11"
format_stamp: "Formatted at 2026-09-25 13:16:19 on dist-test-slave-2d0s"
server_key: "1353e61d4b37a2c6d4749299500ecff3"
server_key_iv: "d7cde1ea2e40323679a8c27701d88129"
server_key_version: "encryptionkey@0"
I20260925 13:16:19.223112 20917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8107809c76d48e78cfb934a90be7a11"
format_stamp: "Formatted at 2026-09-25 13:16:19 on dist-test-slave-2d0s"
server_key: "1353e61d4b37a2c6d4749299500ecff3"
server_key_iv: "d7cde1ea2e40323679a8c27701d88129"
server_key_version: "encryptionkey@0"
I20260925 13:16:19.224406 20917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:16:19.225096 20932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:19.225255 20917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:16:19.225347 20917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "d8107809c76d48e78cfb934a90be7a11"
format_stamp: "Formatted at 2026-09-25 13:16:19 on dist-test-slave-2d0s"
server_key: "1353e61d4b37a2c6d4749299500ecff3"
server_key_iv: "d7cde1ea2e40323679a8c27701d88129"
server_key_version: "encryptionkey@0"
I20260925 13:16:19.225406 20917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:19.238677 20917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:19.240861 20917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:19.240983 20917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:19.244997 20917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45387
I20260925 13:16:19.245059 20984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45387 every 8 connection(s)
I20260925 13:16:19.245440 20917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:19.246415 20985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:19.247319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 20917
I20260925 13:16:19.247480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:19.247671 392 external_mini_cluster.cc:1428] Setting key 3979cc37611d88ecfe5eb8b37a24e5d9
I20260925 13:16:19.249938 20985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11: Bootstrap starting.
I20260925 13:16:19.250716 20985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:19.251035 20985 log.cc:824] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:19.251763 20985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11: No bootstrap required, opened a new log
I20260925 13:16:19.254253 20985 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8107809c76d48e78cfb934a90be7a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260925 13:16:19.254402 20985 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:19.254426 20985 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8107809c76d48e78cfb934a90be7a11, State: Initialized, Role: FOLLOWER
I20260925 13:16:19.254572 20985 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8107809c76d48e78cfb934a90be7a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260925 13:16:19.254660 20985 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:19.254705 20985 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:19.254803 20985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 13:16:19 dist-test-slave-2d0s krb5kdc[20904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:16:19.255487 20985 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8107809c76d48e78cfb934a90be7a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260925 13:16:19.255607 20985 leader_election.cc:302] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8107809c76d48e78cfb934a90be7a11; no voters:
I20260925 13:16:19.255848 20985 leader_election.cc:288] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:19.255894 20990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:19.256021 20990 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [term 1 LEADER]: Becoming Leader. State: Replica: d8107809c76d48e78cfb934a90be7a11, State: Running, Role: LEADER
I20260925 13:16:19.256174 20985 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:19.256510 20990 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8107809c76d48e78cfb934a90be7a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260925 13:16:19.257391 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8107809c76d48e78cfb934a90be7a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8107809c76d48e78cfb934a90be7a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } } }
I20260925 13:16:19.257488 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:19.257623 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8107809c76d48e78cfb934a90be7a11. Latest consensus state: current_term: 1 leader_uuid: "d8107809c76d48e78cfb934a90be7a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8107809c76d48e78cfb934a90be7a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } } }
I20260925 13:16:19.257908 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11 [sys.catalog]: This master's current role is: LEADER
W20260925 13:16:19.258411 21004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:19.258466 21004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:19.258569 21005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:19.258793 21005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:19.259902 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:19.248249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59857 (local address 127.0.0.1:45387)
0925 13:16:19.248468 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:19.248478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:19.248513 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:16:19.249038 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:19.249048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:19.249117 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:19.249282 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:19.250025 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:19.251400 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:19.252253 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:19.252889 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:19.253009 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:19.255812 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:19.255817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:19.255818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:19.255830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:19.258099 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:19.258560 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:19.258564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:19.258565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:19.258701 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:19.258919 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:19.258920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:19.258921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:19.259235 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:19.259394 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:19.259541 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 13:16:19.259693 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260925 13:16:19.260650 21005 catalog_manager.cc:1347] Generated new cluster ID: 0c4f7c70136b4b7691f10ff3456f35a5
I20260925 13:16:19.260700 21005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:19.261775 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:19.270646 21005 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:19.271332 21005 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:19.273382 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:19.263303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48985 (local address 127.0.0.1:45387)
0925 13:16:19.263342 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:19.263344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:19.265894 (+ 2550us) server_negotiation.cc:407] Connection header received
0925 13:16:19.265927 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:19.265929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:19.265941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:19.266017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:19.266657 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:19.267454 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:19.268459 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:19.268784 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:19.268861 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:19.270446 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:19.270448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:19.270448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:19.270456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:19.272189 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:19.272591 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:19.272592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:19.272593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:19.272657 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:19.272855 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:19.272856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:19.272857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:19.272990 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:19.273090 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:19.273190 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 13:16:19.273286 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:19.279695 21005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8107809c76d48e78cfb934a90be7a11: Generated new TSK 0
I20260925 13:16:19.279907 21005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:19.282651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 20917
2026-09-25T13:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:19 dist-test-slave-2d0s krb5kdc[21017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:19 dist-test-slave-2d0s krb5kdc[21017](info): set up 2 sockets
Sep 25 13:16:19 dist-test-slave-2d0s krb5kdc[21017](info): commencing operation
krb5kdc: starting...
W20260925 13:16:20.318106 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:20 dist-test-slave-2d0s krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:20.452443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:20.539213 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:20.539441 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:20.539466 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:20.540603 21030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:20.540643 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:20.540658 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:20.540673 21030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:20.540685 21030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:20.542271 21030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:20.542501 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:20.542692 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:20.548012 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:20.548004 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:20.548007 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:20.548414 21030 server_base.cc:1034] running on GCE node
I20260925 13:16:20.548691 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:20.549055 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:20.550274 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790342180550274 us; error 55 us; skew 500 ppm
Sep 25 13:16:20 dist-test-slave-2d0s krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:20.561091 21030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:20.561653 21030 webserver.cc:460] Webserver started at http://127.0.0.1:38475/ using document root <none> and password file <none>
I20260925 13:16:20.561991 21030 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:20.562083 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:20.562248 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:20.563444 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "956908384a7646fa838423bae67dafc0"
format_stamp: "Formatted at 2026-09-25 13:16:20 on dist-test-slave-2d0s"
server_key: "c91dc1402648465944f90e1de5a81d80"
server_key_iv: "2837eb822fa3273e17f96ac9caf833d6"
server_key_version: "encryptionkey@0"
I20260925 13:16:20.563853 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "956908384a7646fa838423bae67dafc0"
format_stamp: "Formatted at 2026-09-25 13:16:20 on dist-test-slave-2d0s"
server_key: "c91dc1402648465944f90e1de5a81d80"
server_key_iv: "2837eb822fa3273e17f96ac9caf833d6"
server_key_version: "encryptionkey@0"
I20260925 13:16:20.565347 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:20.566295 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:20.566499 21030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:16:20.566622 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "956908384a7646fa838423bae67dafc0"
format_stamp: "Formatted at 2026-09-25 13:16:20 on dist-test-slave-2d0s"
server_key: "c91dc1402648465944f90e1de5a81d80"
server_key_iv: "2837eb822fa3273e17f96ac9caf833d6"
server_key_version: "encryptionkey@0"
I20260925 13:16:20.566708 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:20.599152 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:20.601449 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:20.601634 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:20.605651 21030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44083
I20260925 13:16:20.605743 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44083 every 8 connection(s)
I20260925 13:16:20.606137 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:20.607066 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:20.608752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21030
I20260925 13:16:20.608858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:20.609067 392 external_mini_cluster.cc:1428] Setting key e337eb6a0c626c736ed32437cf8237aa
I20260925 13:16:20.609900 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0: Bootstrap starting.
I20260925 13:16:20.610615 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:20.610968 21098 log.cc:824] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:20.611752 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0: No bootstrap required, opened a new log
I20260925 13:16:20.614382 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956908384a7646fa838423bae67dafc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260925 13:16:20.614595 21098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:20.614646 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956908384a7646fa838423bae67dafc0, State: Initialized, Role: FOLLOWER
I20260925 13:16:20.614791 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956908384a7646fa838423bae67dafc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260925 13:16:20.614898 21098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:20.614966 21098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:20.615044 21098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:20.615231 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:20.609621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53449 (local address 127.0.0.1:44083)
0925 13:16:20.609836 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:20.609845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:20.609872 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:16:20.610365 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:20.610373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:20.610425 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:20.610544 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:20.610853 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:20.611826 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:20.613015 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:20.613460 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:20.613552 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:20.614419 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:20.614423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:20.614423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:20.614598 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:20.614663 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:20.614679 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:20.614819 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:20.614932 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 13:16:20.615004 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":127,"threads_started":1}
I20260925 13:16:20.615823 21098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956908384a7646fa838423bae67dafc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260925 13:16:20.615971 21098 leader_election.cc:302] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 956908384a7646fa838423bae67dafc0; no voters:
I20260925 13:16:20.616168 21098 leader_election.cc:288] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:20.616199 21103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:20.616344 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:20.616390 21103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [term 1 LEADER]: Becoming Leader. State: Replica: 956908384a7646fa838423bae67dafc0, State: Running, Role: LEADER
I20260925 13:16:20.616483 21103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956908384a7646fa838423bae67dafc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260925 13:16:20.616529 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:20.616770 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 956908384a7646fa838423bae67dafc0. Latest consensus state: current_term: 1 leader_uuid: "956908384a7646fa838423bae67dafc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956908384a7646fa838423bae67dafc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } } }
I20260925 13:16:20.616782 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "956908384a7646fa838423bae67dafc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956908384a7646fa838423bae67dafc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } } }
I20260925 13:16:20.616849 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:20.617012 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:20.617087 21108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:20.617640 21108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:20.620863 21108 catalog_manager.cc:1347] Generated new cluster ID: 1d34a4fd7a784535b12e9ae394205ca7
I20260925 13:16:20.620923 21108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:20.626356 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:20.618069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58321 (local address 127.0.0.1:44083)
0925 13:16:20.618104 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:20.618106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:20.618500 (+ 394us) server_negotiation.cc:407] Connection header received
0925 13:16:20.618550 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:20.618551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:20.618561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:20.618649 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:20.618803 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:20.619405 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:20.620142 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:20.620403 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:20.620487 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:20.626010 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:20.626013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:20.626013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:20.626021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:20.626047 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:20.626053 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:20.626143 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:20.626224 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:16:20.626264 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:16:20.628190 21108 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:20.628752 21108 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:20.632562 21108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 956908384a7646fa838423bae67dafc0: Generated new TSK 0
I20260925 13:16:20.632748 21108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:20.641942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21030
2026-09-25T13:16:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:20 dist-test-slave-2d0s krb5kdc[21129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:20 dist-test-slave-2d0s krb5kdc[21129](info): set up 2 sockets
Sep 25 13:16:20 dist-test-slave-2d0s krb5kdc[21129](info): commencing operation
krb5kdc: starting...
W20260925 13:16:21.673907 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:21 dist-test-slave-2d0s krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:21.792726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:21.875553 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:21.875834 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:21.875875 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:21.877115 21142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:21.877197 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:21.877219 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:21.877238 21142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:21.877265 21142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:21.878836 21142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:21.879137 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:21.879379 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:21.884348 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:21.884371 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:21.884352 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:21.884589 21142 server_base.cc:1034] running on GCE node
I20260925 13:16:21.884845 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:21.885205 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:21.886382 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790342181886361 us; error 34 us; skew 500 ppm
Sep 25 13:16:21 dist-test-slave-2d0s krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:21.889055 21142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:21.889525 21142 webserver.cc:460] Webserver started at http://127.0.0.1:34199/ using document root <none> and password file <none>
I20260925 13:16:21.889818 21142 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:21.889894 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:21.890051 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:21.891165 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "e8a16e1dc0c1423596aceb031333e9c3"
format_stamp: "Formatted at 2026-09-25 13:16:21 on dist-test-slave-2d0s"
server_key: "90d01015b8a243dddc69d5024b09faeb"
server_key_iv: "e1d485c4d4a086f61a5ae067d0fc9eec"
server_key_version: "encryptionkey@0"
I20260925 13:16:21.891543 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8a16e1dc0c1423596aceb031333e9c3"
format_stamp: "Formatted at 2026-09-25 13:16:21 on dist-test-slave-2d0s"
server_key: "90d01015b8a243dddc69d5024b09faeb"
server_key_iv: "e1d485c4d4a086f61a5ae067d0fc9eec"
server_key_version: "encryptionkey@0"
I20260925 13:16:21.893038 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:21.893951 21157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:21.894150 21142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:21.894255 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "e8a16e1dc0c1423596aceb031333e9c3"
format_stamp: "Formatted at 2026-09-25 13:16:21 on dist-test-slave-2d0s"
server_key: "90d01015b8a243dddc69d5024b09faeb"
server_key_iv: "e1d485c4d4a086f61a5ae067d0fc9eec"
server_key_version: "encryptionkey@0"
I20260925 13:16:21.894343 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:21.897797 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:21.900267 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:21.900434 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:21.904604 21142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36093
I20260925 13:16:21.904652 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36093 every 8 connection(s)
I20260925 13:16:21.905033 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:21.906172 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:21.908002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21142
I20260925 13:16:21.908135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:21.908317 392 external_mini_cluster.cc:1428] Setting key bafa3a3f928869f7f643ff286123d0c1
I20260925 13:16:21.908901 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3: Bootstrap starting.
I20260925 13:16:21.909569 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:21.909952 21210 log.cc:824] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:21.910832 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3: No bootstrap required, opened a new log
I20260925 13:16:21.911180 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:21.908872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36263 (local address 127.0.0.1:36093)
0925 13:16:21.909062 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:21.909070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:21.909097 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:16:21.909542 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:21.909549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:21.909589 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:21.909858 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:21.910721 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:21.910724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:21.910725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:21.910729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:21.910767 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:21.910778 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:21.910848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:21.910930 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 13:16:21.910976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 13:16:21.912211 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:21.913129 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a16e1dc0c1423596aceb031333e9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260925 13:16:21.913344 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:21.913398 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a16e1dc0c1423596aceb031333e9c3, State: Initialized, Role: FOLLOWER
I20260925 13:16:21.913534 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a16e1dc0c1423596aceb031333e9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260925 13:16:21.913648 21210 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:21.913753 21210 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:21.913805 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:21.914561 21210 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a16e1dc0c1423596aceb031333e9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260925 13:16:21.914695 21210 leader_election.cc:302] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8a16e1dc0c1423596aceb031333e9c3; no voters:
I20260925 13:16:21.914880 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:21.913652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59255 (local address 127.0.0.1:36093)
0925 13:16:21.913710 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:21.913712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:21.913830 (+ 118us) server_negotiation.cc:407] Connection header received
0925 13:16:21.913934 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:21.913934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:21.913940 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:21.914006 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:21.914646 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:21.914647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:21.914648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:21.914652 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:21.914666 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:21.914671 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:21.914719 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:21.914802 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 13:16:21.914817 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:16:21.914896 21210 leader_election.cc:288] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:21.914937 21220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:21.915217 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:21.915275 21220 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8a16e1dc0c1423596aceb031333e9c3, State: Running, Role: LEADER
I20260925 13:16:21.915382 21220 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a16e1dc0c1423596aceb031333e9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260925 13:16:21.915733 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8a16e1dc0c1423596aceb031333e9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a16e1dc0c1423596aceb031333e9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } } }
I20260925 13:16:21.915807 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:21.915982 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a16e1dc0c1423596aceb031333e9c3. Latest consensus state: current_term: 1 leader_uuid: "e8a16e1dc0c1423596aceb031333e9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a16e1dc0c1423596aceb031333e9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } } }
I20260925 13:16:21.916057 21228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:21.916119 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:21.916684 21228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:21.918322 21228 catalog_manager.cc:1347] Generated new cluster ID: b4d229bc1ac04f2d844e294b08de7224
I20260925 13:16:21.918375 21228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:21.931345 21228 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:21.931560 21228 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:21.935758 21228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8a16e1dc0c1423596aceb031333e9c3: Generated new TSK 0
I20260925 13:16:21.935988 21228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:21.939915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21142
2026-09-25T13:16:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:21 dist-test-slave-2d0s krb5kdc[21241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:21 dist-test-slave-2d0s krb5kdc[21241](info): set up 2 sockets
Sep 25 13:16:21 dist-test-slave-2d0s krb5kdc[21241](info): commencing operation
krb5kdc: starting...
W20260925 13:16:22.989914 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:23 dist-test-slave-2d0s krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:23.109133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:23.191962 21254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:23.192231 21254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:23.192256 21254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:23.193523 21254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:23.193572 21254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:23.193588 21254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:23.193600 21254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:23.193652 21254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:23.195214 21254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:23.195470 21254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:23.195705 21254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:23.200598 21259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:23.200719 21254 server_base.cc:1034] running on GCE node
W20260925 13:16:23.200600 21260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:23.200600 21262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:23.201120 21254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:23.201467 21254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:23.202643 21254 hybrid_clock.cc:648] HybridClock initialized: now 1790342183202616 us; error 56 us; skew 500 ppm
Sep 25 13:16:23 dist-test-slave-2d0s krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:23.205102 21254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:23.205607 21254 webserver.cc:460] Webserver started at http://127.0.0.1:39097/ using document root <none> and password file <none>
I20260925 13:16:23.205873 21254 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:23.205943 21254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:23.206108 21254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:23.207217 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa"
format_stamp: "Formatted at 2026-09-25 13:16:23 on dist-test-slave-2d0s"
server_key: "f8a31571087d4a52fc92a2e500a1eca7"
server_key_iv: "19d4a2192f02c2baf8127f520b7aa149"
server_key_version: "encryptionkey@0"
I20260925 13:16:23.207592 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa"
format_stamp: "Formatted at 2026-09-25 13:16:23 on dist-test-slave-2d0s"
server_key: "f8a31571087d4a52fc92a2e500a1eca7"
server_key_iv: "19d4a2192f02c2baf8127f520b7aa149"
server_key_version: "encryptionkey@0"
I20260925 13:16:23.208931 21254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:23.209704 21269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:23.209892 21254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:23.210004 21254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa"
format_stamp: "Formatted at 2026-09-25 13:16:23 on dist-test-slave-2d0s"
server_key: "f8a31571087d4a52fc92a2e500a1eca7"
server_key_iv: "19d4a2192f02c2baf8127f520b7aa149"
server_key_version: "encryptionkey@0"
I20260925 13:16:23.210090 21254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:23.230788 21254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:23.232787 21254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:23.232972 21254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:23.237401 21254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36405
I20260925 13:16:23.237442 21321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36405 every 8 connection(s)
I20260925 13:16:23.237876 21254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:23.238876 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:23.241222 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa: Bootstrap starting.
I20260925 13:16:23.241916 21322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:23.242213 21322 log.cc:824] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:23.242962 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa: No bootstrap required, opened a new log
I20260925 13:16:23.244590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21254
I20260925 13:16:23.244700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:23.244592 21322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260925 13:16:23.244829 21322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:23.244899 21322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5dffc1b5524e4a8065d7102aa0c2aa, State: Initialized, Role: FOLLOWER
I20260925 13:16:23.245014 392 external_mini_cluster.cc:1428] Setting key d2893f5b22576078d6b888cf2a8bc68d
I20260925 13:16:23.245081 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260925 13:16:23.245172 21322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:23.245231 21322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:23.245275 21322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:23.246079 21322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260925 13:16:23.246207 21322 leader_election.cc:302] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b5dffc1b5524e4a8065d7102aa0c2aa; no voters:
I20260925 13:16:23.246371 21322 leader_election.cc:288] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:23.246392 21327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:23.246539 21327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [term 1 LEADER]: Becoming Leader. State: Replica: 6b5dffc1b5524e4a8065d7102aa0c2aa, State: Running, Role: LEADER
I20260925 13:16:23.246639 21322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:23.246671 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260925 13:16:23.247008 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b5dffc1b5524e4a8065d7102aa0c2aa. Latest consensus state: current_term: 1 leader_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } } }
I20260925 13:16:23.247078 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:23.248488 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5dffc1b5524e4a8065d7102aa0c2aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } } }
I20260925 13:16:23.248632 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa [sys.catalog]: This master's current role is: LEADER
W20260925 13:16:23.248739 21343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:23.248816 21343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:23.248884 21331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:23.249063 21331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:23.250594 21331 catalog_manager.cc:1347] Generated new cluster ID: d0670c3707f546d8a039c9e78418ba41
I20260925 13:16:23.250648 21331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:16:23 dist-test-slave-2d0s krb5kdc[21241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:16:23.258421 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:23.245610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:57705 (local address 127.0.0.1:36405)
0925 13:16:23.246277 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:23.246288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:23.246331 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:16:23.246459 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:23.246467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:23.246547 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:23.246714 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:23.247501 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:23.248804 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:23.249955 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:23.250550 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:23.250691 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:23.253634 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:23.253638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:23.253639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:23.253652 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:23.256505 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:23.257020 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:23.257024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:23.257025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:23.257155 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:23.257399 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:23.257400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:23.257401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:23.257777 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:23.257950 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:23.258083 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:16:23.258193 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":556,"threads_started":1}
I20260925 13:16:23.259461 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:23.268263 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:23.260941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:56559 (local address 127.0.0.1:36405)
0925 13:16:23.260977 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:23.260979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:23.261055 (+ 76us) server_negotiation.cc:407] Connection header received
0925 13:16:23.261113 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:23.261114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:23.261124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:23.261191 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:23.261780 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:23.262574 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:23.263553 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:23.263886 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:23.263959 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:23.265360 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:23.265362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:23.265362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:23.265369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:23.267022 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:23.267449 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:23.267450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:23.267450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:23.267514 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:23.267733 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:23.267734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:23.267734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:23.267870 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:23.267973 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:23.268093 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 13:16:23.268159 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:16:23.273634 21331 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:23.274474 21331 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:23.279889 21331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b5dffc1b5524e4a8065d7102aa0c2aa: Generated new TSK 0
I20260925 13:16:23.280117 21331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:23.288728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21254
W20260925 13:16:23.293217 21239 connection.cc:535] client connection to 127.0.0.1:36405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36405): Connection reset by peer (error 104)
2026-09-25T13:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:23 dist-test-slave-2d0s krb5kdc[21353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:23 dist-test-slave-2d0s krb5kdc[21353](info): set up 2 sockets
Sep 25 13:16:23 dist-test-slave-2d0s krb5kdc[21353](info): commencing operation
krb5kdc: starting...
W20260925 13:16:24.333835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:24 dist-test-slave-2d0s krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:24.454191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:24.536084 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:24.536314 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:24.536339 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:24.537570 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:24.537619 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:24.537633 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:24.537647 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:24.537707 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:24.539251 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:24.539454 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:24.539626 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:24.544615 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:24.544646 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:24.544641 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:24.544907 21366 server_base.cc:1034] running on GCE node
I20260925 13:16:24.545113 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:24.545372 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:24.546510 21366 hybrid_clock.cc:648] HybridClock initialized: now 1790342184546498 us; error 30 us; skew 500 ppm
Sep 25 13:16:24 dist-test-slave-2d0s krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:24.549057 21366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:24.549525 21366 webserver.cc:460] Webserver started at http://127.0.0.1:40077/ using document root <none> and password file <none>
I20260925 13:16:24.549798 21366 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:24.549865 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:24.550024 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:24.551123 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "526c58b88c9f40afa8962e4814fc8ee8"
format_stamp: "Formatted at 2026-09-25 13:16:24 on dist-test-slave-2d0s"
server_key: "d3157761d5ef917595b2228f9e4fc872"
server_key_iv: "d64cf94e0ec8cd9018fa9374637f01ab"
server_key_version: "encryptionkey@0"
I20260925 13:16:24.551505 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "526c58b88c9f40afa8962e4814fc8ee8"
format_stamp: "Formatted at 2026-09-25 13:16:24 on dist-test-slave-2d0s"
server_key: "d3157761d5ef917595b2228f9e4fc872"
server_key_iv: "d64cf94e0ec8cd9018fa9374637f01ab"
server_key_version: "encryptionkey@0"
I20260925 13:16:24.552906 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:24.553776 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:24.553962 21366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:24.554061 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "526c58b88c9f40afa8962e4814fc8ee8"
format_stamp: "Formatted at 2026-09-25 13:16:24 on dist-test-slave-2d0s"
server_key: "d3157761d5ef917595b2228f9e4fc872"
server_key_iv: "d64cf94e0ec8cd9018fa9374637f01ab"
server_key_version: "encryptionkey@0"
I20260925 13:16:24.554152 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:24.567912 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:24.570561 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:24.570744 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:24.574687 21366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39603
I20260925 13:16:24.574724 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39603 every 8 connection(s)
I20260925 13:16:24.575090 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:24.575974 21434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:24.578339 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8: Bootstrap starting.
I20260925 13:16:24.578896 21434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:24.579069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21366
I20260925 13:16:24.579123 21434 log.cc:824] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:24.579156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:24.579308 392 external_mini_cluster.cc:1428] Setting key f93f5d4bffc5bb5fbf9808a5b465e258
I20260925 13:16:24.579914 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8: No bootstrap required, opened a new log
I20260925 13:16:24.581862 21434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c58b88c9f40afa8962e4814fc8ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39603 } }
I20260925 13:16:24.582036 21434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:24.582080 21434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c58b88c9f40afa8962e4814fc8ee8, State: Initialized, Role: FOLLOWER
I20260925 13:16:24.582216 21434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c58b88c9f40afa8962e4814fc8ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39603 } }
I20260925 13:16:24.582316 21434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:24.582362 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:24.582417 21434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:24.583009 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c58b88c9f40afa8962e4814fc8ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39603 } }
I20260925 13:16:24.583132 21434 leader_election.cc:302] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526c58b88c9f40afa8962e4814fc8ee8; no voters:
I20260925 13:16:24.583328 21434 leader_election.cc:288] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:24.583384 21439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:24.583526 21439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 526c58b88c9f40afa8962e4814fc8ee8, State: Running, Role: LEADER
I20260925 13:16:24.583607 21434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:24.583611 21439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c58b88c9f40afa8962e4814fc8ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39603 } }
I20260925 13:16:24.583992 21441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 526c58b88c9f40afa8962e4814fc8ee8. Latest consensus state: current_term: 1 leader_uuid: "526c58b88c9f40afa8962e4814fc8ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c58b88c9f40afa8962e4814fc8ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39603 } } }
I20260925 13:16:24.584103 21441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:24.583990 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "526c58b88c9f40afa8962e4814fc8ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c58b88c9f40afa8962e4814fc8ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39603 } } }
I20260925 13:16:24.584278 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:24.584347 21446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:24.584883 21446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:24.587385 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:24.579863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:51043 (local address 127.0.0.1:39603)
0925 13:16:24.580051 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:24.580061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:24.580094 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:16:24.581827 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:24.581837 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:24.581901 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:24.582075 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:24.582316 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:24.583928 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:24.585110 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:24.585747 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:24.585858 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:24.586634 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:24.586638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:24.586638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:24.586881 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:24.586919 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:24.586934 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:24.587019 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:24.587126 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 13:16:24.587156 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260925 13:16:24.588493 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:24.590857 21446 catalog_manager.cc:1347] Generated new cluster ID: c9190ce7674a41918505ff90fc315d3a
I20260925 13:16:24.590919 21446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:24.593994 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:24.590150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:41185 (local address 127.0.0.1:39603)
0925 13:16:24.590326 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:24.590328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:24.590337 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:24.590358 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:24.590358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:24.590369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:24.590434 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:24.590622 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:24.591372 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:24.592405 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:24.592798 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:24.592876 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:24.593647 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:24.593648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:24.593649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:24.593654 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:24.593698 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:24.593703 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:24.593768 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:24.593902 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 13:16:24.593917 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260925 13:16:24.598353 21446 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:24.599084 21446 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:24.612300 21446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 526c58b88c9f40afa8962e4814fc8ee8: Generated new TSK 0
I20260925 13:16:24.612561 21446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:24.620642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21366
W20260925 13:16:24.624948 21351 connection.cc:535] client connection to 127.0.0.1:39603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39603): Connection reset by peer (error 104)
2026-09-25T13:16:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:24 dist-test-slave-2d0s krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:24 dist-test-slave-2d0s krb5kdc[21465](info): set up 2 sockets
Sep 25 13:16:24 dist-test-slave-2d0s krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260925 13:16:25.657943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:25 dist-test-slave-2d0s krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:25.776829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:25.859213 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:25.859453 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:25.859478 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:25.860961 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:25.861027 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:25.861042 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:25.861055 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:25.861068 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:25.862928 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:25.863214 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:25.863457 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:25.868383 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:25.868376 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:25.868420 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:25.868589 21478 server_base.cc:1034] running on GCE node
I20260925 13:16:25.868882 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:25.869257 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:25.870445 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790342185870429 us; error 26 us; skew 500 ppm
Sep 25 13:16:25 dist-test-slave-2d0s krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:25.879477 21478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:25.880004 21478 webserver.cc:460] Webserver started at http://127.0.0.1:44471/ using document root <none> and password file <none>
I20260925 13:16:25.880216 21478 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:25.880260 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:25.880354 21478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:25.881418 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "f65a68ddd4ef42808f944163710da3d2"
format_stamp: "Formatted at 2026-09-25 13:16:25 on dist-test-slave-2d0s"
server_key: "447bc4c15909b42768f03f84fa5472ac"
server_key_iv: "22b2b84975e965ed6461f4214c962377"
server_key_version: "encryptionkey@0"
I20260925 13:16:25.881815 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "f65a68ddd4ef42808f944163710da3d2"
format_stamp: "Formatted at 2026-09-25 13:16:25 on dist-test-slave-2d0s"
server_key: "447bc4c15909b42768f03f84fa5472ac"
server_key_iv: "22b2b84975e965ed6461f4214c962377"
server_key_version: "encryptionkey@0"
I20260925 13:16:25.883179 21478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:25.883908 21493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:25.884085 21478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:25.884184 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "f65a68ddd4ef42808f944163710da3d2"
format_stamp: "Formatted at 2026-09-25 13:16:25 on dist-test-slave-2d0s"
server_key: "447bc4c15909b42768f03f84fa5472ac"
server_key_iv: "22b2b84975e965ed6461f4214c962377"
server_key_version: "encryptionkey@0"
I20260925 13:16:25.884254 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:25.887588 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:25.889590 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:25.889773 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:25.893460 21478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36625
I20260925 13:16:25.893545 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36625 every 8 connection(s)
I20260925 13:16:25.893898 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:25.894956 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:25.897221 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2: Bootstrap starting.
I20260925 13:16:25.897842 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:25.898083 21546 log.cc:824] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:25.898764 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2: No bootstrap required, opened a new log
I20260925 13:16:25.900313 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a68ddd4ef42808f944163710da3d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260925 13:16:25.900454 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:25.900476 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a68ddd4ef42808f944163710da3d2, State: Initialized, Role: FOLLOWER
I20260925 13:16:25.900584 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a68ddd4ef42808f944163710da3d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260925 13:16:25.900672 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:25.900709 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:25.900787 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:25.901363 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a68ddd4ef42808f944163710da3d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260925 13:16:25.901489 21546 leader_election.cc:302] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65a68ddd4ef42808f944163710da3d2; no voters:
I20260925 13:16:25.901742 21546 leader_election.cc:288] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:25.901778 21549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:25.901888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21478
I20260925 13:16:25.902009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:25.902024 21549 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [term 1 LEADER]: Becoming Leader. State: Replica: f65a68ddd4ef42808f944163710da3d2, State: Running, Role: LEADER
I20260925 13:16:25.902036 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:25.902117 21549 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a68ddd4ef42808f944163710da3d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260925 13:16:25.902249 392 external_mini_cluster.cc:1428] Setting key 6e51eeeb73239e0d42da15aed07e5886
I20260925 13:16:25.902429 21549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65a68ddd4ef42808f944163710da3d2. Latest consensus state: current_term: 1 leader_uuid: "f65a68ddd4ef42808f944163710da3d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a68ddd4ef42808f944163710da3d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } } }
I20260925 13:16:25.902506 21549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:25.902594 21551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65a68ddd4ef42808f944163710da3d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65a68ddd4ef42808f944163710da3d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } } }
I20260925 13:16:25.902650 21551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:25.903456 21555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:25.904223 21555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:25.904860 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:25.902895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:46861 (local address 127.0.0.1:36625)
0925 13:16:25.903056 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:25.903066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:25.903099 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:16:25.903222 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:25.903231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:25.903266 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:25.903407 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:25.904356 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:25.904359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:25.904360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:25.904364 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:25.904412 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:25.904427 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:25.904489 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:25.904599 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 13:16:25.904642 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260925 13:16:25.905802 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:25.906046 21555 catalog_manager.cc:1347] Generated new cluster ID: 0a76fca7f4bc4c4d84dfde2520d312ff
I20260925 13:16:25.906112 21555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:25.908628 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:25.907315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:53911 (local address 127.0.0.1:36625)
0925 13:16:25.907348 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:25.907350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:25.907603 (+ 253us) server_negotiation.cc:407] Connection header received
0925 13:16:25.907672 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:25.907673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:25.907689 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:25.907770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:25.908408 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:25.908410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:25.908410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:25.908415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:25.908431 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:25.908436 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:25.908500 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:25.908552 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 13:16:25.908565 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:16:25.922544 21555 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:25.922778 21555 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:25.931290 21555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65a68ddd4ef42808f944163710da3d2: Generated new TSK 0
I20260925 13:16:25.931520 21555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:25.935847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21478
2026-09-25T13:16:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:25 dist-test-slave-2d0s krb5kdc[21577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:25 dist-test-slave-2d0s krb5kdc[21577](info): set up 2 sockets
Sep 25 13:16:25 dist-test-slave-2d0s krb5kdc[21577](info): commencing operation
krb5kdc: starting...
W20260925 13:16:26.973938 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:27 dist-test-slave-2d0s krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:27.094640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:27.182682 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:27.182921 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:27.182947 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:27.184212 21590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:27.184265 21590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:27.184286 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:27.184333 21590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:27.184365 21590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:27.185972 21590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:27.186292 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:27.186545 21590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:27.191641 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:27.191668 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:27.191694 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:27.191910 21590 server_base.cc:1034] running on GCE node
I20260925 13:16:27.192191 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:27.192541 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:27.193770 21590 hybrid_clock.cc:648] HybridClock initialized: now 1790342187193756 us; error 67 us; skew 500 ppm
Sep 25 13:16:27 dist-test-slave-2d0s krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:27.196699 21590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:27.197237 21590 webserver.cc:460] Webserver started at http://127.0.0.1:34123/ using document root <none> and password file <none>
I20260925 13:16:27.197489 21590 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:27.197561 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:27.197768 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:27.198905 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "89c0925c989b43a1afed4bee7f9cc567"
format_stamp: "Formatted at 2026-09-25 13:16:27 on dist-test-slave-2d0s"
server_key: "6bfdaddec7e4cdf0014b7f729c4f38c2"
server_key_iv: "f7e7c2a1ab6da3ce11ac434b00049192"
server_key_version: "encryptionkey@0"
I20260925 13:16:27.199309 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "89c0925c989b43a1afed4bee7f9cc567"
format_stamp: "Formatted at 2026-09-25 13:16:27 on dist-test-slave-2d0s"
server_key: "6bfdaddec7e4cdf0014b7f729c4f38c2"
server_key_iv: "f7e7c2a1ab6da3ce11ac434b00049192"
server_key_version: "encryptionkey@0"
I20260925 13:16:27.200735 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:16:27.201548 21605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:27.201771 21590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:27.201882 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "89c0925c989b43a1afed4bee7f9cc567"
format_stamp: "Formatted at 2026-09-25 13:16:27 on dist-test-slave-2d0s"
server_key: "6bfdaddec7e4cdf0014b7f729c4f38c2"
server_key_iv: "f7e7c2a1ab6da3ce11ac434b00049192"
server_key_version: "encryptionkey@0"
I20260925 13:16:27.201964 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:27.210563 21590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:27.212522 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:27.212646 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:27.216670 21590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43615
I20260925 13:16:27.216723 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43615 every 8 connection(s)
I20260925 13:16:27.217109 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:27.218101 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:27.219782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21590
I20260925 13:16:27.219921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:27.220121 392 external_mini_cluster.cc:1428] Setting key 41d787f4edcee7da2b615558b66512e8
I20260925 13:16:27.220916 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567: Bootstrap starting.
I20260925 13:16:27.221522 21658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:27.221920 21658 log.cc:824] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:27.222673 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567: No bootstrap required, opened a new log
I20260925 13:16:27.224869 21658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c0925c989b43a1afed4bee7f9cc567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43615 } }
I20260925 13:16:27.225009 21658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:27.225031 21658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c0925c989b43a1afed4bee7f9cc567, State: Initialized, Role: FOLLOWER
I20260925 13:16:27.225205 21658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c0925c989b43a1afed4bee7f9cc567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43615 } }
I20260925 13:16:27.225292 21658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:27.225349 21658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:27.225428 21658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:27.226126 21658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c0925c989b43a1afed4bee7f9cc567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43615 } }
I20260925 13:16:27.226253 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89c0925c989b43a1afed4bee7f9cc567; no voters:
I20260925 13:16:27.226465 21658 leader_election.cc:288] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:27.226493 21663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:27.226610 21663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [term 1 LEADER]: Becoming Leader. State: Replica: 89c0925c989b43a1afed4bee7f9cc567, State: Running, Role: LEADER
I20260925 13:16:27.226753 21658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:27.226747 21663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c0925c989b43a1afed4bee7f9cc567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43615 } }
I20260925 13:16:27.227223 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89c0925c989b43a1afed4bee7f9cc567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c0925c989b43a1afed4bee7f9cc567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43615 } } }
I20260925 13:16:27.227466 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [sys.catalog]: This master's current role is: LEADER
Sep 25 13:16:27 dist-test-slave-2d0s krb5kdc[21577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:16:27.227391 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c0925c989b43a1afed4bee7f9cc567. Latest consensus state: current_term: 1 leader_uuid: "89c0925c989b43a1afed4bee7f9cc567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c0925c989b43a1afed4bee7f9cc567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43615 } } }
I20260925 13:16:27.227900 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:27.228101 21672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:27.228717 21672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:27.230633 21672 catalog_manager.cc:1347] Generated new cluster ID: d8ed864e637a444695aa6fba7dd30c0d
I20260925 13:16:27.230686 21672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:27.232228 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:27.220685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:51971 (local address 127.0.0.1:43615)
0925 13:16:27.220883 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:27.220892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:27.220926 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:16:27.221409 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:27.221416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:27.221468 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:27.221587 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:27.222465 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:27.223404 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:27.224609 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:27.225099 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:27.225224 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:27.228132 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:27.228137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:27.228137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:27.228153 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:27.230327 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:27.230876 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:27.230879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:27.230880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:27.230992 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:27.231250 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:27.231251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:27.231252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:27.231571 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:27.231733 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:27.231877 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 13:16:27.231988 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260925 13:16:27.233529 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:27.238024 21672 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:27.238556 21672 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:27.244135 21672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c0925c989b43a1afed4bee7f9cc567: Generated new TSK 0
I20260925 13:16:27.244386 21672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:27.250924 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:27.235394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:58733 (local address 127.0.0.1:43615)
0925 13:16:27.235436 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:27.235438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:27.237428 (+ 1990us) server_negotiation.cc:407] Connection header received
0925 13:16:27.237487 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:27.237488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:27.237501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:27.237623 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:27.244656 (+ 7033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:27.245470 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:27.246289 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:27.246625 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:27.246713 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:27.248102 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:27.248104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:27.248104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:27.248115 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:27.249769 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:27.250125 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:27.250126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:27.250126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:27.250192 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:27.250407 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:27.250407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:27.250408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:27.250545 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:27.250643 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:27.250753 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 13:16:27.250821 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:16:27.252899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21590
W20260925 13:16:27.259053 21575 connection.cc:535] client connection to 127.0.0.1:43615 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43615): Connection reset by peer (error 104)
2026-09-25T13:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:27 dist-test-slave-2d0s krb5kdc[21689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:27 dist-test-slave-2d0s krb5kdc[21689](info): set up 2 sockets
Sep 25 13:16:27 dist-test-slave-2d0s krb5kdc[21689](info): commencing operation
krb5kdc: starting...
W20260925 13:16:28.285792 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:28 dist-test-slave-2d0s krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:28.402784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:28.484697 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:28.484918 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:28.484942 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:28.486241 21702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:28.486287 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:28.486302 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:28.486315 21702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:28.486351 21702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:28.487859 21702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:28.488112 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:28.488363 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:16:28.493237 21702 server_base.cc:1034] running on GCE node
W20260925 13:16:28.493220 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:28.493207 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:28.493216 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:28.493624 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:28.493985 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:28.495131 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790342188495118 us; error 31 us; skew 500 ppm
Sep 25 13:16:28 dist-test-slave-2d0s krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:28.497742 21702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:28.498172 21702 webserver.cc:460] Webserver started at http://127.0.0.1:40123/ using document root <none> and password file <none>
I20260925 13:16:28.498420 21702 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:28.498466 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:28.498631 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:28.499725 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "8301e902e24c4e979568540ccd90548c"
format_stamp: "Formatted at 2026-09-25 13:16:28 on dist-test-slave-2d0s"
server_key: "af0a10313788e0fa20e8f7c206d41f8f"
server_key_iv: "850f5772171d531c15160606774488fa"
server_key_version: "encryptionkey@0"
I20260925 13:16:28.500100 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "8301e902e24c4e979568540ccd90548c"
format_stamp: "Formatted at 2026-09-25 13:16:28 on dist-test-slave-2d0s"
server_key: "af0a10313788e0fa20e8f7c206d41f8f"
server_key_iv: "850f5772171d531c15160606774488fa"
server_key_version: "encryptionkey@0"
I20260925 13:16:28.501426 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:28.502223 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:28.502393 21702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:28.502494 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "8301e902e24c4e979568540ccd90548c"
format_stamp: "Formatted at 2026-09-25 13:16:28 on dist-test-slave-2d0s"
server_key: "af0a10313788e0fa20e8f7c206d41f8f"
server_key_iv: "850f5772171d531c15160606774488fa"
server_key_version: "encryptionkey@0"
I20260925 13:16:28.502575 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:28.528987 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:28.531422 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:28.531541 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:28.535629 21702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46155
I20260925 13:16:28.535686 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46155 every 8 connection(s)
I20260925 13:16:28.536048 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:28.537112 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:28.537950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21702
I20260925 13:16:28.538096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:28.538311 392 external_mini_cluster.cc:1428] Setting key 85203a1b1da2cad00ac2dde82cfe35a5
I20260925 13:16:28.540567 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c: Bootstrap starting.
I20260925 13:16:28.541302 21770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:28.541615 21770 log.cc:824] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:28.542400 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c: No bootstrap required, opened a new log
I20260925 13:16:28.544488 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:28.538892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:37513 (local address 127.0.0.1:46155)
0925 13:16:28.539094 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:28.539104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:28.539137 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:16:28.539694 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:28.539704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:28.539774 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:28.539928 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:28.540101 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:28.541436 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:28.542291 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:28.542912 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:28.543025 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:28.543727 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:28.543732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:28.543732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:28.543947 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:28.543992 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:28.544011 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:28.544103 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:28.544184 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:16:28.544218 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260925 13:16:28.544761 21770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8301e902e24c4e979568540ccd90548c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260925 13:16:28.544939 21770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:28.544973 21770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8301e902e24c4e979568540ccd90548c, State: Initialized, Role: FOLLOWER
I20260925 13:16:28.545105 21770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8301e902e24c4e979568540ccd90548c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260925 13:16:28.545192 21770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:28.545235 21770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:28.545290 21770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:28.545583 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:28.546033 21770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8301e902e24c4e979568540ccd90548c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260925 13:16:28.546159 21770 leader_election.cc:302] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8301e902e24c4e979568540ccd90548c; no voters:
I20260925 13:16:28.546388 21770 leader_election.cc:288] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:28.546418 21775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:28.546541 21775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [term 1 LEADER]: Becoming Leader. State: Replica: 8301e902e24c4e979568540ccd90548c, State: Running, Role: LEADER
I20260925 13:16:28.546669 21770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:28.546701 21775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8301e902e24c4e979568540ccd90548c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260925 13:16:28.547077 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8301e902e24c4e979568540ccd90548c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8301e902e24c4e979568540ccd90548c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } } }
I20260925 13:16:28.547220 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:28.547415 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8301e902e24c4e979568540ccd90548c. Latest consensus state: current_term: 1 leader_uuid: "8301e902e24c4e979568540ccd90548c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8301e902e24c4e979568540ccd90548c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } } }
I20260925 13:16:28.547511 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:28.547643 21788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:28.548156 21788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:28.550076 21788 catalog_manager.cc:1347] Generated new cluster ID: c52e18e95d274a18a7f108f878b86a78
I20260925 13:16:28.550127 21788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:28.550858 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:28.547014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:39779 (local address 127.0.0.1:46155)
0925 13:16:28.547051 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:28.547053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:28.547178 (+ 125us) server_negotiation.cc:407] Connection header received
0925 13:16:28.547223 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:28.547224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:28.547235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:28.547291 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:28.547433 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:28.548189 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:28.549270 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:28.549555 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:28.549616 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:28.550471 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:28.550472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:28.550472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:28.550476 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:28.550492 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:28.550508 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:28.550579 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:28.550751 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 13:16:28.550762 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19}
I20260925 13:16:28.560851 21788 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:28.561635 21788 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:28.565968 21788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8301e902e24c4e979568540ccd90548c: Generated new TSK 0
I20260925 13:16:28.566176 21788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:28.569831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21702
W20260925 13:16:28.574489 21687 connection.cc:535] client connection to 127.0.0.1:46155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46155): Connection reset by peer (error 104)
2026-09-25T13:16:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:28 dist-test-slave-2d0s krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:28 dist-test-slave-2d0s krb5kdc[21801](info): set up 2 sockets
Sep 25 13:16:28 dist-test-slave-2d0s krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260925 13:16:29.605930 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:29 dist-test-slave-2d0s krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:16:29.726662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:29.811101 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:29.811348 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:29.811375 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:29.812647 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:29.812700 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:29.812716 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:29.812728 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:29.812741 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:29.814425 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:29.814713 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:29.814978 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:16:29.820086 21814 server_base.cc:1034] running on GCE node
W20260925 13:16:29.820034 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:29.820031 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:29.820053 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:29.820535 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:29.820814 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:29.821954 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790342189821939 us; error 45 us; skew 500 ppm
Sep 25 13:16:29 dist-test-slave-2d0s krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:29.824627 21814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:16:29.825120 21814 webserver.cc:460] Webserver started at http://127.0.0.1:37971/ using document root <none> and password file <none>
I20260925 13:16:29.825381 21814 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:29.825451 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:29.825615 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:29.826766 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "3ab2f18088df4b3788970d2a2da07b78"
format_stamp: "Formatted at 2026-09-25 13:16:29 on dist-test-slave-2d0s"
server_key: "fc095fed9a99e476410b70570813ea52"
server_key_iv: "a365d74b552eb074edd92ff7b4f9f6c3"
server_key_version: "encryptionkey@0"
I20260925 13:16:29.827138 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ab2f18088df4b3788970d2a2da07b78"
format_stamp: "Formatted at 2026-09-25 13:16:29 on dist-test-slave-2d0s"
server_key: "fc095fed9a99e476410b70570813ea52"
server_key_iv: "a365d74b552eb074edd92ff7b4f9f6c3"
server_key_version: "encryptionkey@0"
I20260925 13:16:29.828544 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:16:29.829331 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:29.829568 21814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:29.829702 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "3ab2f18088df4b3788970d2a2da07b78"
format_stamp: "Formatted at 2026-09-25 13:16:29 on dist-test-slave-2d0s"
server_key: "fc095fed9a99e476410b70570813ea52"
server_key_iv: "a365d74b552eb074edd92ff7b4f9f6c3"
server_key_version: "encryptionkey@0"
I20260925 13:16:29.829795 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:29.833189 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:29.835373 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:29.835541 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:29.839834 21814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40161
I20260925 13:16:29.839865 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40161 every 8 connection(s)
I20260925 13:16:29.840222 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:29.841198 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:29.841837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21814
I20260925 13:16:29.841967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:29.842151 392 external_mini_cluster.cc:1428] Setting key d62375c7b0b3ce5c6b215a7d2239c078
I20260925 13:16:29.844486 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78: Bootstrap starting.
I20260925 13:16:29.844718 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:29.842709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:51137 (local address 127.0.0.1:40161)
0925 13:16:29.842899 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:29.842907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:29.842934 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:16:29.843344 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:29.843350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:29.843375 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:29.843499 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:29.844201 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:29.844204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:29.844205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:29.844209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:29.844246 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:29.844256 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:29.844313 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:29.844404 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 13:16:29.844465 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 13:16:29.845292 21882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:29.845652 21882 log.cc:824] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:29.845971 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:29.846393 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78: No bootstrap required, opened a new log
I20260925 13:16:29.848526 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:29.847477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:50731 (local address 127.0.0.1:40161)
0925 13:16:29.847520 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:29.847522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:29.847607 (+ 85us) server_negotiation.cc:407] Connection header received
0925 13:16:29.847681 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:29.847681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:29.847686 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:29.847730 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:29.848300 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:29.848301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:29.848301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:29.848305 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:29.848317 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:29.848321 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:29.848365 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:29.848427 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 13:16:29.848438 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:16:29.848897 21882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab2f18088df4b3788970d2a2da07b78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260925 13:16:29.849064 21882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:29.849121 21882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab2f18088df4b3788970d2a2da07b78, State: Initialized, Role: FOLLOWER
I20260925 13:16:29.849268 21882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab2f18088df4b3788970d2a2da07b78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260925 13:16:29.849380 21882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:29.849426 21882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:29.849475 21882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:29.850152 21882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab2f18088df4b3788970d2a2da07b78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260925 13:16:29.850281 21882 leader_election.cc:302] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ab2f18088df4b3788970d2a2da07b78; no voters:
I20260925 13:16:29.850474 21882 leader_election.cc:288] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:29.850553 21892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:29.850723 21892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab2f18088df4b3788970d2a2da07b78, State: Running, Role: LEADER
I20260925 13:16:29.850756 21882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:29.850811 21892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab2f18088df4b3788970d2a2da07b78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260925 13:16:29.851208 21894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ab2f18088df4b3788970d2a2da07b78. Latest consensus state: current_term: 1 leader_uuid: "3ab2f18088df4b3788970d2a2da07b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab2f18088df4b3788970d2a2da07b78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } } }
I20260925 13:16:29.851290 21894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:29.851186 21893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ab2f18088df4b3788970d2a2da07b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab2f18088df4b3788970d2a2da07b78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } } }
I20260925 13:16:29.851421 21893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:29.851519 21900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:29.852011 21900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:29.853549 21900 catalog_manager.cc:1347] Generated new cluster ID: dc65305135ef4facafdde8fdea8fe2dc
I20260925 13:16:29.853609 21900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:29.865619 21900 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:29.865854 21900 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:29.869974 21900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ab2f18088df4b3788970d2a2da07b78: Generated new TSK 0
I20260925 13:16:29.870224 21900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:29.871901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21814
2026-09-25T13:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:29 dist-test-slave-2d0s krb5kdc[21913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:29 dist-test-slave-2d0s krb5kdc[21913](info): set up 2 sockets
Sep 25 13:16:29 dist-test-slave-2d0s krb5kdc[21913](info): commencing operation
krb5kdc: starting...
W20260925 13:16:30.913898 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:30 dist-test-slave-2d0s krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:31.034682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35207
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:31.120298 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:31.120587 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:31.120630 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:31.121999 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:31.122071 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:31.122097 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:31.122129 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:31.122156 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:31.123818 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:31.124143 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:31.124394 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:16:31.129868 21926 server_base.cc:1034] running on GCE node
W20260925 13:16:31.129778 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:31.129787 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:31.129774 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:31.130378 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:31.130692 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:31.131843 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790342191131828 us; error 27 us; skew 500 ppm
Sep 25 13:16:31 dist-test-slave-2d0s krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:31.139985 21926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:31.140538 21926 webserver.cc:460] Webserver started at http://127.0.98.62:38751/ using document root <none> and password file <none>
I20260925 13:16:31.140775 21926 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:31.140843 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:31.141000 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:31.142196 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "ce06cdc5c0f34436915e7b6ffe732f42"
format_stamp: "Formatted at 2026-09-25 13:16:31 on dist-test-slave-2d0s"
server_key: "061d24fe5d2e992c4da7139fb7863eb8"
server_key_iv: "ad135a2e80cdf922069c2fa92e6d3adf"
server_key_version: "encryptionkey@0"
I20260925 13:16:31.142572 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce06cdc5c0f34436915e7b6ffe732f42"
format_stamp: "Formatted at 2026-09-25 13:16:31 on dist-test-slave-2d0s"
server_key: "061d24fe5d2e992c4da7139fb7863eb8"
server_key_iv: "ad135a2e80cdf922069c2fa92e6d3adf"
server_key_version: "encryptionkey@0"
I20260925 13:16:31.143998 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:31.144761 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:31.144966 21926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:31.145056 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "ce06cdc5c0f34436915e7b6ffe732f42"
format_stamp: "Formatted at 2026-09-25 13:16:31 on dist-test-slave-2d0s"
server_key: "061d24fe5d2e992c4da7139fb7863eb8"
server_key_iv: "ad135a2e80cdf922069c2fa92e6d3adf"
server_key_version: "encryptionkey@0"
I20260925 13:16:31.145107 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:31.158776 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:31.160902 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:31.161062 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:31.165478 21926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35207
I20260925 13:16:31.165557 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35207 every 8 connection(s)
I20260925 13:16:31.166029 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:31.166986 21994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:31.169391 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42: Bootstrap starting.
I20260925 13:16:31.170105 21994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:31.170204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 21926
I20260925 13:16:31.170347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:31.170421 21994 log.cc:824] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:31.170571 392 external_mini_cluster.cc:1428] Setting key 2c370ed47704b306678d39b59dac1492
I20260925 13:16:31.171350 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42: No bootstrap required, opened a new log
I20260925 13:16:31.173792 21994 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35207 } }
I20260925 13:16:31.174007 21994 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:31.174048 21994 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce06cdc5c0f34436915e7b6ffe732f42, State: Initialized, Role: FOLLOWER
I20260925 13:16:31.174192 21994 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35207 } }
I20260925 13:16:31.174278 21994 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:31.174316 21994 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:31.174367 21994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:31.175176 21994 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35207 } }
I20260925 13:16:31.175314 21994 leader_election.cc:302] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce06cdc5c0f34436915e7b6ffe732f42; no voters:
I20260925 13:16:31.175570 21994 leader_election.cc:288] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:31.175606 21999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:31.175906 21999 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [term 1 LEADER]: Becoming Leader. State: Replica: ce06cdc5c0f34436915e7b6ffe732f42, State: Running, Role: LEADER
I20260925 13:16:31.175961 21994 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:31.176033 21999 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35207 } }
I20260925 13:16:31.176486 21999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35207 } } }
I20260925 13:16:31.176581 21999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:31.176717 21999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce06cdc5c0f34436915e7b6ffe732f42. Latest consensus state: current_term: 1 leader_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce06cdc5c0f34436915e7b6ffe732f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35207 } } }
I20260925 13:16:31.176777 21999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42 [sys.catalog]: This master's current role is: LEADER
W20260925 13:16:31.178206 22014 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:31.178272 22014 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:31.178323 22007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 13:16:31 dist-test-slave-2d0s krb5kdc[21913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:31.178574 22007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:31.180546 22007 catalog_manager.cc:1347] Generated new cluster ID: 8fa94d45825542c6b83bc7a14979eba9
I20260925 13:16:31.180609 22007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:31.182749 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:31.171150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33991 (local address 127.0.98.62:35207)
0925 13:16:31.171354 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:31.171363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:31.171403 (+ 40us) server_negotiation.cc:407] Connection header received
0925 13:16:31.171906 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:31.171914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:31.171963 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:31.172118 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:31.172769 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:31.173654 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:31.174936 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:31.175480 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:31.175632 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:31.179032 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:31.179037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:31.179037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:31.179048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:31.181089 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:31.181534 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:31.181537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:31.181537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:31.181618 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:31.181922 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:31.181923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:31.181923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:31.182166 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:31.182338 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:31.182462 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 13:16:31.182579 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260925 13:16:31.186960 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:31.196710 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:31.190070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40609 (local address 127.0.98.62:35207)
0925 13:16:31.190135 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:31.190138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:31.190266 (+ 128us) server_negotiation.cc:407] Connection header received
0925 13:16:31.190344 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:31.190345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:31.190356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:31.190412 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:31.190986 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:31.191567 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:31.192286 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:31.192514 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:31.192602 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:31.193999 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:31.194001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:31.194001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:31.194009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:31.195481 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:31.195871 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:31.195872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:31.195873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:31.195935 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:31.196165 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:31.196166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:31.196166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:31.196291 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:31.196427 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:31.196526 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 13:16:31.196592 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:16:31.211320 22007 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:31.211913 22007 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:31.214955 22007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce06cdc5c0f34436915e7b6ffe732f42: Generated new TSK 0
I20260925 13:16:31.215184 22007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:31.218727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 21926
W20260925 13:16:31.223482 21911 connection.cc:535] client connection to 127.0.98.62:35207 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35207): Connection reset by peer (error 104)
2026-09-25T13:16:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:31 dist-test-slave-2d0s krb5kdc[22025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:31 dist-test-slave-2d0s krb5kdc[22025](info): set up 2 sockets
Sep 25 13:16:31 dist-test-slave-2d0s krb5kdc[22025](info): commencing operation
krb5kdc: starting...
W20260925 13:16:32.277911 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:32 dist-test-slave-2d0s krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:32.409480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:32.495790 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:32.496040 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:32.496071 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:32.497442 22038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:32.497509 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:32.497524 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:32.497536 22038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:32.497556 22038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:32.499323 22038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:32.499655 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:32.499925 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:32.505203 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:32.505203 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:32.505227 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:32.505460 22038 server_base.cc:1034] running on GCE node
I20260925 13:16:32.505836 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:32.506170 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:32.507313 22038 hybrid_clock.cc:648] HybridClock initialized: now 1790342192507298 us; error 36 us; skew 500 ppm
Sep 25 13:16:32 dist-test-slave-2d0s krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:32.510098 22038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:32.510577 22038 webserver.cc:460] Webserver started at http://127.0.98.62:39237/ using document root <none> and password file <none>
I20260925 13:16:32.510835 22038 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:32.510922 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:32.511075 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:32.512185 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "6e3145835391421c886f0fb84725396d"
format_stamp: "Formatted at 2026-09-25 13:16:32 on dist-test-slave-2d0s"
server_key: "29435600d65f46e73d210b0b89116aba"
server_key_iv: "305e7fea4312a1ba484ed4b04fa337e7"
server_key_version: "encryptionkey@0"
I20260925 13:16:32.512591 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e3145835391421c886f0fb84725396d"
format_stamp: "Formatted at 2026-09-25 13:16:32 on dist-test-slave-2d0s"
server_key: "29435600d65f46e73d210b0b89116aba"
server_key_iv: "305e7fea4312a1ba484ed4b04fa337e7"
server_key_version: "encryptionkey@0"
I20260925 13:16:32.514051 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:16:32.514889 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:32.515070 22038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:32.515169 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "6e3145835391421c886f0fb84725396d"
format_stamp: "Formatted at 2026-09-25 13:16:32 on dist-test-slave-2d0s"
server_key: "29435600d65f46e73d210b0b89116aba"
server_key_iv: "305e7fea4312a1ba484ed4b04fa337e7"
server_key_version: "encryptionkey@0"
I20260925 13:16:32.515259 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:32.535676 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:32.538014 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:32.538237 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:32.542742 22038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35881
I20260925 13:16:32.542797 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35881 every 8 connection(s)
I20260925 13:16:32.543213 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:32.544162 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:32.545454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22038
I20260925 13:16:32.545600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:32.545831 392 external_mini_cluster.cc:1428] Setting key 03697c2afc756ccd170b2121a33b4090
I20260925 13:16:32.547257 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d: Bootstrap starting.
I20260925 13:16:32.547921 22106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:32.548225 22106 log.cc:824] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:32.548981 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d: No bootstrap required, opened a new log
I20260925 13:16:32.551225 22106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3145835391421c886f0fb84725396d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260925 13:16:32.551414 22106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:32.551466 22106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3145835391421c886f0fb84725396d, State: Initialized, Role: FOLLOWER
I20260925 13:16:32.551641 22106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3145835391421c886f0fb84725396d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260925 13:16:32.551755 22106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:32.551841 22106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:32.551920 22106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:32.552026 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:32.546456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58243 (local address 127.0.98.62:35881)
0925 13:16:32.546667 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:32.546676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:32.546703 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:16:32.547155 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:32.547162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:32.547215 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:32.547379 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:32.547650 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:32.548731 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:32.549699 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:32.550148 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:32.550277 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:32.551183 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:32.551188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:32.551189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:32.551370 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:32.551403 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:32.551415 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:32.551530 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:32.551689 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 13:16:32.551726 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260925 13:16:32.552838 22106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3145835391421c886f0fb84725396d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260925 13:16:32.552986 22106 leader_election.cc:302] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e3145835391421c886f0fb84725396d; no voters:
I20260925 13:16:32.553120 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:32.553182 22106 leader_election.cc:288] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:32.553227 22111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:32.553395 22111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [term 1 LEADER]: Becoming Leader. State: Replica: 6e3145835391421c886f0fb84725396d, State: Running, Role: LEADER
I20260925 13:16:32.553467 22106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:32.553541 22111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3145835391421c886f0fb84725396d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260925 13:16:32.553983 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e3145835391421c886f0fb84725396d. Latest consensus state: current_term: 1 leader_uuid: "6e3145835391421c886f0fb84725396d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3145835391421c886f0fb84725396d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } } }
I20260925 13:16:32.553990 22113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e3145835391421c886f0fb84725396d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3145835391421c886f0fb84725396d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } } }
I20260925 13:16:32.554181 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:32.554236 22113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:32.554522 22124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:32.555207 22124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:32.557049 22133 mvcc.cc:205] Tried to move back new op lower bound from 7333241620710772736 to 7333241620700020736. Current Snapshot: MvccSnapshot[applied={T|T < 7333241620710772736}]
I20260925 13:16:32.557548 22124 catalog_manager.cc:1347] Generated new cluster ID: c42b365812e34728a6fd8a58f6817731
I20260925 13:16:32.557612 22124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:32.559291 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:32.554804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45665 (local address 127.0.98.62:35881)
0925 13:16:32.554846 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:32.554849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:32.555132 (+ 283us) server_negotiation.cc:407] Connection header received
0925 13:16:32.555223 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:32.555224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:32.555237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:32.555303 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:32.555522 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:32.556329 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:32.557596 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:32.557982 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:32.558086 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:32.558982 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:32.558984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:32.558985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:32.558990 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:32.559012 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:32.559019 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:32.559104 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:32.559185 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:16:32.559198 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:32.577764 22124 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:32.578673 22124 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:32.587926 22124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e3145835391421c886f0fb84725396d: Generated new TSK 0
I20260925 13:16:32.588186 22124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:32.591493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22038
W20260925 13:16:32.596452 22023 connection.cc:535] client connection to 127.0.98.62:35881 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35881): Connection reset by peer (error 104)
2026-09-25T13:16:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:32 dist-test-slave-2d0s krb5kdc[22137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:32 dist-test-slave-2d0s krb5kdc[22137](info): set up 2 sockets
Sep 25 13:16:32 dist-test-slave-2d0s krb5kdc[22137](info): commencing operation
krb5kdc: starting...
W20260925 13:16:33.629892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:33 dist-test-slave-2d0s krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:33.752138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:33.835453 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:33.835752 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:33.835793 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:33.837066 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:33.837150 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:33.837175 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:33.837195 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:33.837229 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:33.838887 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:33.839218 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:33.839473 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:33.844455 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:33.844471 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:33.844462 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:33.844580 22150 server_base.cc:1034] running on GCE node
I20260925 13:16:33.845022 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:33.845360 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:33.846532 22150 hybrid_clock.cc:648] HybridClock initialized: now 1790342193846521 us; error 45 us; skew 500 ppm
Sep 25 13:16:33 dist-test-slave-2d0s krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:33.849121 22150 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:33.849654 22150 webserver.cc:460] Webserver started at http://127.0.98.62:41399/ using document root <none> and password file <none>
I20260925 13:16:33.849960 22150 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:33.850033 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:33.850190 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:33.851339 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "fc78ccb5e6744af98e56848eaec8a221"
format_stamp: "Formatted at 2026-09-25 13:16:33 on dist-test-slave-2d0s"
server_key: "41b14cff70065f70c1950109cb0f11f0"
server_key_iv: "e4d6c434cf614092009a8098ed068589"
server_key_version: "encryptionkey@0"
I20260925 13:16:33.851737 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc78ccb5e6744af98e56848eaec8a221"
format_stamp: "Formatted at 2026-09-25 13:16:33 on dist-test-slave-2d0s"
server_key: "41b14cff70065f70c1950109cb0f11f0"
server_key_iv: "e4d6c434cf614092009a8098ed068589"
server_key_version: "encryptionkey@0"
I20260925 13:16:33.853191 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:33.854045 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:33.854230 22150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:16:33.854341 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "fc78ccb5e6744af98e56848eaec8a221"
format_stamp: "Formatted at 2026-09-25 13:16:33 on dist-test-slave-2d0s"
server_key: "41b14cff70065f70c1950109cb0f11f0"
server_key_iv: "e4d6c434cf614092009a8098ed068589"
server_key_version: "encryptionkey@0"
I20260925 13:16:33.854416 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:33.857890 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:33.859925 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:33.860103 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:33.864457 22150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45705
I20260925 13:16:33.864490 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45705 every 8 connection(s)
I20260925 13:16:33.864864 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:33.866007 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:33.867139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22150
I20260925 13:16:33.867288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:33.867472 392 external_mini_cluster.cc:1428] Setting key 6b9b66d55a2c755aebbf2b23e1253bda
I20260925 13:16:33.868906 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221: Bootstrap starting.
I20260925 13:16:33.869709 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:33.870038 22218 log.cc:824] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:33.870348 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:33.868091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50081 (local address 127.0.98.62:45705)
0925 13:16:33.868301 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:33.868312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:33.868349 (+ 37us) server_negotiation.cc:407] Connection header received
0925 13:16:33.868868 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:33.868878 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:33.868916 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:33.869091 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:33.869878 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:33.869882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:33.869882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:33.869887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:33.869935 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:33.869947 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:33.870063 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:33.870105 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 13:16:33.870152 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260925 13:16:33.870913 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221: No bootstrap required, opened a new log
I20260925 13:16:33.871433 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:33.873229 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc78ccb5e6744af98e56848eaec8a221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260925 13:16:33.873404 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:33.873445 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc78ccb5e6744af98e56848eaec8a221, State: Initialized, Role: FOLLOWER
I20260925 13:16:33.873569 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc78ccb5e6744af98e56848eaec8a221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260925 13:16:33.873695 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:33.873735 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:33.873821 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:33.874104 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:33.872858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37457 (local address 127.0.98.62:45705)
0925 13:16:33.872894 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:33.872895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:33.873050 (+ 155us) server_negotiation.cc:407] Connection header received
0925 13:16:33.873130 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:33.873131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:33.873137 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:33.873211 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:33.873895 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:33.873896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:33.873896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:33.873901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:33.873914 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:33.873919 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:33.873974 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:33.874021 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 13:16:33.874036 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:16:33.874677 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc78ccb5e6744af98e56848eaec8a221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260925 13:16:33.874809 22218 leader_election.cc:302] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc78ccb5e6744af98e56848eaec8a221; no voters:
I20260925 13:16:33.874990 22218 leader_election.cc:288] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:33.875034 22228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:33.875170 22228 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [term 1 LEADER]: Becoming Leader. State: Replica: fc78ccb5e6744af98e56848eaec8a221, State: Running, Role: LEADER
I20260925 13:16:33.875270 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:33.875305 22228 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc78ccb5e6744af98e56848eaec8a221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260925 13:16:33.875726 22230 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc78ccb5e6744af98e56848eaec8a221. Latest consensus state: current_term: 1 leader_uuid: "fc78ccb5e6744af98e56848eaec8a221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc78ccb5e6744af98e56848eaec8a221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } } }
I20260925 13:16:33.875731 22229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc78ccb5e6744af98e56848eaec8a221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc78ccb5e6744af98e56848eaec8a221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } } }
I20260925 13:16:33.875859 22230 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:33.875874 22229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:33.876143 22237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:33.876693 22237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:33.878468 22237 catalog_manager.cc:1347] Generated new cluster ID: b24395809f964c9489a84af3d320167f
I20260925 13:16:33.878548 22237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:33.882802 22237 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:33.883006 22237 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:33.887990 22237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc78ccb5e6744af98e56848eaec8a221: Generated new TSK 0
I20260925 13:16:33.888218 22237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:33.890849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22150
2026-09-25T13:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:33 dist-test-slave-2d0s krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:33 dist-test-slave-2d0s krb5kdc[22249](info): set up 2 sockets
Sep 25 13:16:33 dist-test-slave-2d0s krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260925 13:16:34.945969 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:35 dist-test-slave-2d0s krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:35.065073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:35.148118 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:35.148337 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:35.148360 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:35.149504 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:35.149539 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:35.149554 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:35.149566 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:35.149580 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:35.151307 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:35.151576 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:35.151821 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:35.156893 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:35.157044 22262 server_base.cc:1034] running on GCE node
W20260925 13:16:35.156929 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:35.156929 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:35.157405 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:35.157753 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:35.158927 22262 hybrid_clock.cc:648] HybridClock initialized: now 1790342195158912 us; error 27 us; skew 500 ppm
Sep 25 13:16:35 dist-test-slave-2d0s krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:35.161536 22262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:35.162102 22262 webserver.cc:460] Webserver started at http://127.0.98.62:33133/ using document root <none> and password file <none>
I20260925 13:16:35.162354 22262 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:35.162426 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:35.162580 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:35.163687 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "ecb6dbbd5dc74912888a1a684d69fb4d"
format_stamp: "Formatted at 2026-09-25 13:16:35 on dist-test-slave-2d0s"
server_key: "8929fab4b94f08ae9c65d066f90ea687"
server_key_iv: "b457935dff34361657940dbca8cb88c5"
server_key_version: "encryptionkey@0"
I20260925 13:16:35.164063 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecb6dbbd5dc74912888a1a684d69fb4d"
format_stamp: "Formatted at 2026-09-25 13:16:35 on dist-test-slave-2d0s"
server_key: "8929fab4b94f08ae9c65d066f90ea687"
server_key_iv: "b457935dff34361657940dbca8cb88c5"
server_key_version: "encryptionkey@0"
I20260925 13:16:35.165447 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:16:35.166354 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:35.166566 22262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:16:35.166671 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "ecb6dbbd5dc74912888a1a684d69fb4d"
format_stamp: "Formatted at 2026-09-25 13:16:35 on dist-test-slave-2d0s"
server_key: "8929fab4b94f08ae9c65d066f90ea687"
server_key_iv: "b457935dff34361657940dbca8cb88c5"
server_key_version: "encryptionkey@0"
I20260925 13:16:35.166759 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:35.183449 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:35.186069 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:35.186249 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:35.190454 22262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41491
I20260925 13:16:35.190505 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41491 every 8 connection(s)
I20260925 13:16:35.190882 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:35.191828 22330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:35.194250 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d: Bootstrap starting.
I20260925 13:16:35.194887 22330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:35.195186 22330 log.cc:824] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:35.195971 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d: No bootstrap required, opened a new log
I20260925 13:16:35.197696 22330 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41491 } }
I20260925 13:16:35.197853 22330 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:35.197876 22330 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb6dbbd5dc74912888a1a684d69fb4d, State: Initialized, Role: FOLLOWER
I20260925 13:16:35.198050 22330 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41491 } }
I20260925 13:16:35.198118 22330 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:35.198141 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:35.198215 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:35.198843 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41491 } }
I20260925 13:16:35.198961 22330 leader_election.cc:302] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecb6dbbd5dc74912888a1a684d69fb4d; no voters:
I20260925 13:16:35.199162 22330 leader_election.cc:288] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:35.199215 22333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:35.199415 22333 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [term 1 LEADER]: Becoming Leader. State: Replica: ecb6dbbd5dc74912888a1a684d69fb4d, State: Running, Role: LEADER
I20260925 13:16:35.199481 22330 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:35.199604 22333 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41491 } }
I20260925 13:16:35.200449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22262
I20260925 13:16:35.200315 22335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecb6dbbd5dc74912888a1a684d69fb4d. Latest consensus state: current_term: 1 leader_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41491 } } }
I20260925 13:16:35.200580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:35.200632 22335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:35.200816 392 external_mini_cluster.cc:1428] Setting key a303d09e93652284b64ffa4cd3248cad
I20260925 13:16:35.200204 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb6dbbd5dc74912888a1a684d69fb4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41491 } } }
I20260925 13:16:35.200910 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:35.200907 22346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:35.201531 22346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:35.203423 22346 catalog_manager.cc:1347] Generated new cluster ID: 07b1dff5b90641db9465e83e5dc41bff
I20260925 13:16:35.203483 22346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:16:35 dist-test-slave-2d0s krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:35.213454 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:35.201373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41907 (local address 127.0.98.62:41491)
0925 13:16:35.201694 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:35.201705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:35.201740 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:16:35.201874 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:35.201881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:35.201940 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:35.202079 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:35.203090 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:35.204517 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:35.205612 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:35.206319 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:35.206460 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:35.209488 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:35.209493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:35.209493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:35.209508 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:35.211635 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:35.212080 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:35.212083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:35.212084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:35.212211 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:35.212446 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:35.212446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:35.212447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:35.212762 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:35.212893 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:35.213049 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 13:16:35.213198 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260925 13:16:35.214537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:35.217784 22346 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:35.218497 22346 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:35.223357 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:35.215974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49219 (local address 127.0.98.62:41491)
0925 13:16:35.216014 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:35.216016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:35.216178 (+ 162us) server_negotiation.cc:407] Connection header received
0925 13:16:35.216224 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:35.216225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:35.216253 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:35.216304 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:35.216944 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:35.217767 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:35.218738 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:35.219064 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:35.219136 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:35.220522 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:35.220524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:35.220524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:35.220531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:35.222197 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:35.222578 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:35.222579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:35.222579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:35.222641 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:35.222832 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:35.222833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:35.222833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:35.222966 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:35.223066 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:35.223166 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 13:16:35.223262 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:35.225899 22346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecb6dbbd5dc74912888a1a684d69fb4d: Generated new TSK 0
I20260925 13:16:35.226097 22346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:35.228495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22262
2026-09-25T13:16:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:35 dist-test-slave-2d0s krb5kdc[22361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:35 dist-test-slave-2d0s krb5kdc[22361](info): set up 2 sockets
Sep 25 13:16:35 dist-test-slave-2d0s krb5kdc[22361](info): commencing operation
krb5kdc: starting...
W20260925 13:16:36.261950 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:36 dist-test-slave-2d0s krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:36.381225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:36.469767 22374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:36.470021 22374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:36.470047 22374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:36.471553 22374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:36.471621 22374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:36.471637 22374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:36.471649 22374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:36.471662 22374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:36.473387 22374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:36.473727 22374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:36.474021 22374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:36.479267 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:36.479264 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:36.479264 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:36.479605 22374 server_base.cc:1034] running on GCE node
I20260925 13:16:36.479861 22374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:36.480155 22374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:36.481386 22374 hybrid_clock.cc:648] HybridClock initialized: now 1790342196481343 us; error 46 us; skew 500 ppm
Sep 25 13:16:36 dist-test-slave-2d0s krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:36.490155 22374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:36.490751 22374 webserver.cc:460] Webserver started at http://127.0.98.62:34759/ using document root <none> and password file <none>
I20260925 13:16:36.491010 22374 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:36.491087 22374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:36.491246 22374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:36.492578 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "facdbfbaf9b04400a8fa6d62c77231cd"
format_stamp: "Formatted at 2026-09-25 13:16:36 on dist-test-slave-2d0s"
server_key: "d014553f93e0eceee0424b98b8b8e164"
server_key_iv: "2e7df65efb226902203489ff20e83618"
server_key_version: "encryptionkey@0"
I20260925 13:16:36.493002 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "facdbfbaf9b04400a8fa6d62c77231cd"
format_stamp: "Formatted at 2026-09-25 13:16:36 on dist-test-slave-2d0s"
server_key: "d014553f93e0eceee0424b98b8b8e164"
server_key_iv: "2e7df65efb226902203489ff20e83618"
server_key_version: "encryptionkey@0"
I20260925 13:16:36.494618 22374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 13:16:36.495554 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:36.495802 22374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:36.495921 22374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "facdbfbaf9b04400a8fa6d62c77231cd"
format_stamp: "Formatted at 2026-09-25 13:16:36 on dist-test-slave-2d0s"
server_key: "d014553f93e0eceee0424b98b8b8e164"
server_key_iv: "2e7df65efb226902203489ff20e83618"
server_key_version: "encryptionkey@0"
I20260925 13:16:36.496016 22374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:36.505002 22374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:36.507400 22374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:36.507598 22374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:36.511999 22374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40357
I20260925 13:16:36.512045 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40357 every 8 connection(s)
I20260925 13:16:36.512445 22374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:36.513427 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:36.516222 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd: Bootstrap starting.
I20260925 13:16:36.517006 22442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:36.517014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22374
I20260925 13:16:36.517128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:36.517367 392 external_mini_cluster.cc:1428] Setting key fa3e7f15b9cac6c4ca6861b29292cb4e
I20260925 13:16:36.517419 22442 log.cc:824] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:36.518519 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd: No bootstrap required, opened a new log
I20260925 13:16:36.520998 22442 raft_consensus.cc:348] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40357 } }
I20260925 13:16:36.521220 22442 raft_consensus.cc:374] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:36.521263 22442 raft_consensus.cc:729] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facdbfbaf9b04400a8fa6d62c77231cd, State: Initialized, Role: FOLLOWER
I20260925 13:16:36.521409 22442 consensus_queue.cc:260] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40357 } }
I20260925 13:16:36.521584 22442 raft_consensus.cc:388] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:36.521790 22442 raft_consensus.cc:482] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:36.521907 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:36.522856 22442 raft_consensus.cc:504] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40357 } }
I20260925 13:16:36.523006 22442 leader_election.cc:302] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: facdbfbaf9b04400a8fa6d62c77231cd; no voters:
I20260925 13:16:36.523219 22442 leader_election.cc:288] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:36.523268 22447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:36.523449 22447 raft_consensus.cc:686] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [term 1 LEADER]: Becoming Leader. State: Replica: facdbfbaf9b04400a8fa6d62c77231cd, State: Running, Role: LEADER
I20260925 13:16:36.523518 22442 sys_catalog.cc:564] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:36.523603 22447 consensus_queue.cc:237] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40357 } }
I20260925 13:16:36.524039 22448 sys_catalog.cc:455] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40357 } } }
I20260925 13:16:36.524154 22448 sys_catalog.cc:458] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:36.524237 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:36.517995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53025 (local address 127.0.98.62:40357)
0925 13:16:36.518191 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:36.518200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:36.518231 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:16:36.518828 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:36.518836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:36.518892 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:36.519023 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:36.519334 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:36.520238 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:36.521459 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:36.522120 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:36.522209 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:36.523053 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:36.523058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:36.523059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:36.523344 (+ 285us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:36.523393 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:36.523412 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:36.523816 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:36.523853 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:16:36.523929 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260925 13:16:36.524740 22456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:36.524933 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader facdbfbaf9b04400a8fa6d62c77231cd. Latest consensus state: current_term: 1 leader_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "facdbfbaf9b04400a8fa6d62c77231cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40357 } } }
I20260925 13:16:36.525178 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:36.525218 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:36.525458 22456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:36.527469 22456 catalog_manager.cc:1347] Generated new cluster ID: ddac38d20acd4794a2ab28186360676e
I20260925 13:16:36.527540 22456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:36.530835 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:36.526908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41583 (local address 127.0.98.62:40357)
0925 13:16:36.526953 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:36.526955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:36.527091 (+ 136us) server_negotiation.cc:407] Connection header received
0925 13:16:36.527182 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:36.527184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:36.527197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:36.527264 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:36.527436 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:36.528206 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:36.529303 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:36.529630 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:36.529762 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:36.530558 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:36.530560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:36.530560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:36.530566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:36.530585 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:36.530590 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:36.530651 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:36.530716 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 13:16:36.530743 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:16:36.540752 22456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:36.541563 22456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:36.546048 22456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P facdbfbaf9b04400a8fa6d62c77231cd: Generated new TSK 0
I20260925 13:16:36.546279 22456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:36.554252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22374
2026-09-25T13:16:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:36 dist-test-slave-2d0s krb5kdc[22473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:36 dist-test-slave-2d0s krb5kdc[22473](info): set up 2 sockets
Sep 25 13:16:36 dist-test-slave-2d0s krb5kdc[22473](info): commencing operation
krb5kdc: starting...
W20260925 13:16:37.585991 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:37 dist-test-slave-2d0s krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:37.708561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43719
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:37.796139 22486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:37.796427 22486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:37.796469 22486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:37.797793 22486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:37.797887 22486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:37.797910 22486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:37.797930 22486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:37.797991 22486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:37.799624 22486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:37.799957 22486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:37.800230 22486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:37.805346 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:37.805351 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:37.805372 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:37.806192 22486 server_base.cc:1034] running on GCE node
I20260925 13:16:37.806432 22486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:37.806802 22486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:37.807976 22486 hybrid_clock.cc:648] HybridClock initialized: now 1790342197807953 us; error 38 us; skew 500 ppm
Sep 25 13:16:37 dist-test-slave-2d0s krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:37.810678 22486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:37.811159 22486 webserver.cc:460] Webserver started at http://127.0.98.62:44649/ using document root <none> and password file <none>
I20260925 13:16:37.811400 22486 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:37.811471 22486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:37.811630 22486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:37.812808 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "39c99e2470db4ae29061ce2dfaae4cd2"
format_stamp: "Formatted at 2026-09-25 13:16:37 on dist-test-slave-2d0s"
server_key: "cd5ccfcae8119482109450c366c0e780"
server_key_iv: "d9e17502c51fb98d2970b8c134326009"
server_key_version: "encryptionkey@0"
I20260925 13:16:37.813198 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "39c99e2470db4ae29061ce2dfaae4cd2"
format_stamp: "Formatted at 2026-09-25 13:16:37 on dist-test-slave-2d0s"
server_key: "cd5ccfcae8119482109450c366c0e780"
server_key_iv: "d9e17502c51fb98d2970b8c134326009"
server_key_version: "encryptionkey@0"
I20260925 13:16:37.814754 22486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:37.815671 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:37.815850 22486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:37.815960 22486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "39c99e2470db4ae29061ce2dfaae4cd2"
format_stamp: "Formatted at 2026-09-25 13:16:37 on dist-test-slave-2d0s"
server_key: "cd5ccfcae8119482109450c366c0e780"
server_key_iv: "d9e17502c51fb98d2970b8c134326009"
server_key_version: "encryptionkey@0"
I20260925 13:16:37.816051 22486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:37.819648 22486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:37.822041 22486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:37.822233 22486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:37.826514 22486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43719
I20260925 13:16:37.826556 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43719 every 8 connection(s)
I20260925 13:16:37.826932 22486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:37.828055 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:37.830643 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2: Bootstrap starting.
I20260925 13:16:37.831279 22554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:37.831578 22554 log.cc:824] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:37.832393 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2: No bootstrap required, opened a new log
I20260925 13:16:37.833923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22486
I20260925 13:16:37.834055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:37.834268 392 external_mini_cluster.cc:1428] Setting key e776e5e0c23bbea83abe7ae94ceacdaa
I20260925 13:16:37.834273 22554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260925 13:16:37.834481 22554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:16:37.834529 22554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c99e2470db4ae29061ce2dfaae4cd2, State: Initialized, Role: FOLLOWER
I20260925 13:16:37.834718 22554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260925 13:16:37.834812 22554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:16:37.834877 22554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:16:37.834944 22554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:37.835546 22554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260925 13:16:37.835646 22554 leader_election.cc:302] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c99e2470db4ae29061ce2dfaae4cd2; no voters:
I20260925 13:16:37.835836 22554 leader_election.cc:288] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:16:37.835879 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:37.836160 22554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:37.836424 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 39c99e2470db4ae29061ce2dfaae4cd2, State: Running, Role: LEADER
I20260925 13:16:37.836586 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } }
I20260925 13:16:37.837069 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } } }
I20260925 13:16:37.837193 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:37.837249 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:37.834834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49097 (local address 127.0.98.62:43719)
0925 13:16:37.835195 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:37.835203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:37.835229 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:16:37.835560 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:37.835566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:37.835591 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:37.835800 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:37.836601 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:37.836606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:37.836606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:37.836612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:37.836661 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:37.836676 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:37.836726 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:37.836940 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 13:16:37.836991 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260925 13:16:37.837086 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c99e2470db4ae29061ce2dfaae4cd2. Latest consensus state: current_term: 1 leader_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c99e2470db4ae29061ce2dfaae4cd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43719 } } }
I20260925 13:16:37.837428 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2 [sys.catalog]: This master's current role is: LEADER
W20260925 13:16:37.838347 22574 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:37.838405 22574 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:37.838476 22571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:37.838527 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:37.838646 22571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:37.840571 22571 catalog_manager.cc:1347] Generated new cluster ID: 798da493fd6b469c8e01866264afc8e1
I20260925 13:16:37.840636 22571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:37.841305 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:37.840125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49905 (local address 127.0.98.62:43719)
0925 13:16:37.840166 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:37.840168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:37.840268 (+ 100us) server_negotiation.cc:407] Connection header received
0925 13:16:37.840335 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:37.840336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:37.840343 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:37.840404 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:37.841047 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:37.841048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:37.841048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:16:37.841053 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:37.841068 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:16:37.841072 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:37.841105 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:37.841215 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 13:16:37.841231 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:16:37.857734 22571 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:37.857959 22571 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:37.868459 22571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c99e2470db4ae29061ce2dfaae4cd2: Generated new TSK 0
I20260925 13:16:37.868697 22571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:16:37.877127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22486
2026-09-25T13:16:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1322 ms)
[----------] 18 tests from AuthTokenIssuingTest (23876 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:37 dist-test-slave-2d0s krb5kdc[22585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:37 dist-test-slave-2d0s krb5kdc[22585](info): set up 2 sockets
Sep 25 13:16:37 dist-test-slave-2d0s krb5kdc[22585](info): commencing operation
krb5kdc: starting...
W20260925 13:16:38.917996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:38 dist-test-slave-2d0s krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:39.040210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42339
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42339,127.0.98.61:40697,127.0.98.60:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:39.124156 22598 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:16:39.124424 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:39.124444 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:39.124465 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:39.125635 22598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:39.125708 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:39.125725 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:39.125737 22598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:39.125749 22598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:39.127488 22598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42339,127.0.98.61:40697,127.0.98.60:44197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:39.127750 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:39.127943 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:39.132901 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:39.132901 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:39.133118 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:39.133189 22598 server_base.cc:1034] running on GCE node
I20260925 13:16:39.133440 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:39.133814 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:39.134986 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790342199134975 us; error 33 us; skew 500 ppm
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:39.137842 22598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:39.138396 22598 webserver.cc:460] Webserver started at http://127.0.98.62:35947/ using document root <none> and password file <none>
I20260925 13:16:39.138645 22598 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:39.138716 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:39.138867 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:39.139984 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "e31074e287e54cc0b2fea031c4a176b9"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "75d930c1a142e8f5b540cc56707b31d2"
server_key_iv: "e56b6ae4ef576a7c0024c9a0d9a9a7a4"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.140377 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "e31074e287e54cc0b2fea031c4a176b9"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "75d930c1a142e8f5b540cc56707b31d2"
server_key_iv: "e56b6ae4ef576a7c0024c9a0d9a9a7a4"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.141798 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:39.142580 22613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:39.142752 22598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:39.142855 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "e31074e287e54cc0b2fea031c4a176b9"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "75d930c1a142e8f5b540cc56707b31d2"
server_key_iv: "e56b6ae4ef576a7c0024c9a0d9a9a7a4"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.142939 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:39.151471 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:39.153815 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:39.153986 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:39.158416 22598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42339
I20260925 13:16:39.158447 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42339 every 8 connection(s)
I20260925 13:16:39.158830 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:39.159876 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:39.161546 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:39.165410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22598
I20260925 13:16:39.165512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:39.165740 392 external_mini_cluster.cc:1428] Setting key 5ff31aeb8b68c2df9f6ae67c5a511bf8
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:39.172525 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.162383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42339 (local address 127.0.0.1:44114)
0925 13:16:39.162651 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.162712 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.163254 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.163561 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.163564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.163886 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.164052 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.164057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.165115 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.165116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.166044 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.166047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.166585 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.166586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.166651 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.167281 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.167290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.168966 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.170945 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.170952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.170954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.171160 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.171416 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.171418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.171418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.171513 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.171892 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.171897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.171961 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.172071 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.172326 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":98,"thread_start_us":121,"threads_started":1}
I20260925 13:16:39.172525 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.162465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.0.98.62:42339)
0925 13:16:39.162663 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.162673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.162828 (+ 155us) server_negotiation.cc:407] Connection header received
0925 13:16:39.163434 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.163435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.163453 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.163546 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.164164 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.165002 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.166142 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.166466 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.166605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.169076 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.169084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.169084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.169091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.170838 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.171258 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.171261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.171262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.171348 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.171579 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.171580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.171581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.171819 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.171920 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.172108 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.172261 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260925 13:16:39.173750 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:39.174173 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.174040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33582)
0925 13:16:39.174090 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.174136 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:16:39.174309 22614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
I20260925 13:16:39.174705 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.174596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33590)
0925 13:16:39.174647 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.174683 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 13:16:39.174845 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40697: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 13:16:39.190948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40697
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42339,127.0.98.61:40697,127.0.98.60:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:16:39.224135 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } attempt: 1
I20260925 13:16:39.224702 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.224541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33594)
0925 13:16:39.224613 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.224657 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 13:16:39.225112 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.225012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33600)
0925 13:16:39.225058 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.225085 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:16:39.225253 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40697: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
W20260925 13:16:39.274501 22673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:16:39.274744 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:39.274761 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:39.274787 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:39.276142 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:39.276188 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:39.276211 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:39.276260 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:39.276281 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:39.277968 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42339,127.0.98.61:40697,127.0.98.60:44197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:39.278244 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:39.278483 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:39.283510 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:39.283501 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:39.283597 22673 server_base.cc:1034] running on GCE node
W20260925 13:16:39.283478 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:39.283866 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:39.284173 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:39.285331 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790342199285304 us; error 50 us; skew 500 ppm
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:39.287994 22673 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 13:16:39.288401 22673 webserver.cc:460] Webserver started at http://127.0.98.61:44027/ using document root <none> and password file <none>
I20260925 13:16:39.288586 22673 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:39.288625 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:39.288723 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:39.289887 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data/instance:
uuid: "d8970b0c51814fe59f5b9981c4d0f8e7"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "fbe9c10a3d9eecee6dffcde0aa797a30"
server_key_iv: "f1871cc99273b92a8be64f60189f1f88"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.290203 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal/instance:
uuid: "d8970b0c51814fe59f5b9981c4d0f8e7"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "fbe9c10a3d9eecee6dffcde0aa797a30"
server_key_iv: "f1871cc99273b92a8be64f60189f1f88"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.291517 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:39.292299 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:39.292508 22673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:39.292598 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal
uuid: "d8970b0c51814fe59f5b9981c4d0f8e7"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "fbe9c10a3d9eecee6dffcde0aa797a30"
server_key_iv: "f1871cc99273b92a8be64f60189f1f88"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.292658 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:39.293395 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } attempt: 2
I20260925 13:16:39.293876 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.293721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33604)
0925 13:16:39.293793 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.293839 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:16:39.294268 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.294174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33610)
0925 13:16:39.294219 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.294245 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:16:39.294382 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40697: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40697: connect: Connection refused (error 111)
I20260925 13:16:39.318761 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:39.320891 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:39.321043 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:39.324949 22673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40697
I20260925 13:16:39.325006 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40697 every 8 connection(s)
I20260925 13:16:39.325390 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:16:39.326483 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:39.326483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22673
I20260925 13:16:39.326575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-1/wal/instance
I20260925 13:16:39.326789 392 external_mini_cluster.cc:1428] Setting key d1c3eb2017b4c6c447d5e7ca8053501a
I20260925 13:16:39.328840 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:39.340672 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.329657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.0.98.62:42339)
0925 13:16:39.329750 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.329753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.329972 (+ 219us) server_negotiation.cc:407] Connection header received
0925 13:16:39.330530 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.330532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.330548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.330628 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.331237 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.332104 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.333088 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.333430 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.333508 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.337681 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.337683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.337683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.337692 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.339241 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.339658 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.339659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.339660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.339732 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.339990 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.339991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.339992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.340117 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.340191 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.340483 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.340572 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 13:16:39.340860 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.329616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42339 (local address 127.0.0.1:44124)
0925 13:16:39.329833 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.329875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.330417 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.330695 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.330699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.330945 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.331137 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.331141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.332213 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.332213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.332999 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.333001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.333530 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.333530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.333635 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.334267 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.334276 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.337564 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.339333 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.339339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.339341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.339562 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.339808 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.339809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.339809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.339917 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.340201 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.340209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.340345 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.340451 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.340615 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":141,"threads_started":1}
I20260925 13:16:39.341436 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:16:39.349409 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.341802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.0.98.61:40697)
0925 13:16:39.341960 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.341965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.342005 (+ 40us) server_negotiation.cc:407] Connection header received
0925 13:16:39.342064 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.342067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.342079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.342144 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.342419 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.343119 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.343808 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.344125 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.344206 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.346697 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.346705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.346706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.346712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.348145 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.348548 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.348552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.348552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.348611 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.348822 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.348823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.348824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.348973 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.349072 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.349180 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.349282 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260925 13:16:39.349412 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.341769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33620)
0925 13:16:39.341947 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.341958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.342007 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.342142 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.342143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.342290 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.342338 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.342339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.343207 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.343208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.343714 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.343715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.344214 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.344215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.344281 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.344725 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.344735 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.346593 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.348236 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.348237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.348238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.348459 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.348687 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.348688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.348688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.348742 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.349049 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.349050 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.349088 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.349153 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.349326 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 13:16:39.350244 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:39.350631 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.350492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34796)
0925 13:16:39.350546 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.350600 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:16:39.350620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44197
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42339,127.0.98.61:40697,127.0.98.60:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:39.350805 22690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
I20260925 13:16:39.351244 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.351124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34798)
0925 13:16:39.351182 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.351214 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 13:16:39.351382 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
I20260925 13:16:39.385568 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } attempt: 3
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:16:39.393365 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.386027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33622)
0925 13:16:39.386111 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.386126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.386198 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.386349 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.386350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.386501 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.386567 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.386568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.387556 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.387556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.388076 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.388077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.388599 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.388600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.388685 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.389212 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.389216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.390631 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.392426 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.392427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.392427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.392584 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.392784 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.392785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.392785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.392835 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.393106 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.393107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.393150 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.393210 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.393288 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 13:16:39.393435 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.386107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.0.98.61:40697)
0925 13:16:39.386183 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.386185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.386198 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:16:39.386263 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.386265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.386278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.386331 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.386642 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.387441 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.388170 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.388495 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.388572 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.390725 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.390726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.390727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.390735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.392339 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.392656 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.392657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.392657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.392719 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.392900 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.392900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.392901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.393036 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.393101 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.393255 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.393339 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:16:39.393931 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:39.394263 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.394161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34810)
0925 13:16:39.394210 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.394242 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:16:39.394642 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.394564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34816)
0925 13:16:39.394605 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.394628 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:16:39.394767 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
I20260925 13:16:39.400631 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } attempt: 1
I20260925 13:16:39.401166 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.401026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34824)
0925 13:16:39.401096 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.401137 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 13:16:39.401571 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.401490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34836)
0925 13:16:39.401530 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.401555 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 13:16:39.401719 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
I20260925 13:16:39.425942 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } attempt: 1
I20260925 13:16:39.426442 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.426282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34838)
0925 13:16:39.426359 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.426407 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 13:16:39.426802 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.426692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34848)
0925 13:16:39.426743 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.426772 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:16:39.426903 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
W20260925 13:16:39.438063 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:16:39.438295 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:39.438314 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:39.438338 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:39.439488 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:39.439527 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:39.439543 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:39.439554 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:39.439566 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:39.441017 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42339,127.0.98.61:40697,127.0.98.60:44197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:39.441226 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:39.441416 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:39.446384 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:39.446408 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:39.446400 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:39.446619 22748 server_base.cc:1034] running on GCE node
I20260925 13:16:39.446913 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:39.447216 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:39.448362 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790342199448349 us; error 28 us; skew 500 ppm
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:39.451021 22748 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 13:16:39.451490 22748 webserver.cc:460] Webserver started at http://127.0.98.60:36021/ using document root <none> and password file <none>
I20260925 13:16:39.451738 22748 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:39.451788 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:39.451965 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:39.453052 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data/instance:
uuid: "63da2a59dc8444669133f3c128474846"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "98f11168c1ed6731c519b60d271859a4"
server_key_iv: "eaaa98763d46492657c67b9a0f9bb575"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.453433 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal/instance:
uuid: "63da2a59dc8444669133f3c128474846"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "98f11168c1ed6731c519b60d271859a4"
server_key_iv: "eaaa98763d46492657c67b9a0f9bb575"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.454865 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:39.455612 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:39.455798 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:16:39.455899 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal
uuid: "63da2a59dc8444669133f3c128474846"
format_stamp: "Formatted at 2026-09-25 13:16:39 on dist-test-slave-2d0s"
server_key: "98f11168c1ed6731c519b60d271859a4"
server_key_iv: "eaaa98763d46492657c67b9a0f9bb575"
server_key_version: "encryptionkey@0"
I20260925 13:16:39.455986 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:39.469875 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } attempt: 2
I20260925 13:16:39.470397 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.470247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34856)
0925 13:16:39.470321 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.470366 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 13:16:39.470769 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.470668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34868)
0925 13:16:39.470715 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.470742 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:16:39.470891 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
I20260925 13:16:39.498761 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:39.500721 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:39.500895 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:39.502039 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } attempt: 2
I20260925 13:16:39.502512 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.502380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34880)
0925 13:16:39.502445 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.502483 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 13:16:39.502941 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.502831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34896)
0925 13:16:39.502878 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.502910 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:16:39.503059 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44197: connect: Connection refused (error 111)
I20260925 13:16:39.505209 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44197
I20260925 13:16:39.505241 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44197 every 8 connection(s)
I20260925 13:16:39.505599 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:16:39.506299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22748
I20260925 13:16:39.506381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790341932215442-392-0/minicluster-data/master-2/wal/instance
I20260925 13:16:39.506593 392 external_mini_cluster.cc:1428] Setting key b2db3b42ebc74d1bef339c270d32738e
I20260925 13:16:39.506726 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:39.509151 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:39.517882 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.509910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.0.98.62:42339)
0925 13:16:39.509975 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.509977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.510093 (+ 116us) server_negotiation.cc:407] Connection header received
0925 13:16:39.510151 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.510153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.510163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.510231 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.510785 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.511350 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.512262 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.512505 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.512558 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.514982 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.514983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.514984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.514992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.516579 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.517052 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.517053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.517053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.517100 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.517280 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.517281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.517281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.517412 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.517478 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.517738 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.517816 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:16:39.517997 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.507095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.98.60:44197)
0925 13:16:39.507294 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.507304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.507336 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:16:39.507958 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.507966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.508031 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.508202 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.508781 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.510258 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.511192 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.511706 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.511787 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.514474 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.514482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.514482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.514491 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.516230 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.516668 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.516671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.516671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.516803 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.516996 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.516997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.516998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.517339 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.517433 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.517581 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.517745 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 13:16:39.517997 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.509829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42339 (local address 127.0.0.1:44134)
0925 13:16:39.509993 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.510031 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.510104 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.510250 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.510252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.510610 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.510704 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.510706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.511438 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.511438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.512173 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.512175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.512609 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.512611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.512698 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.513141 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.513150 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.514916 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.516661 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.516665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.516668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.516984 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.517162 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.517163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.517163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.517219 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.517482 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.517487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.517576 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.517701 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.517793 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260925 13:16:39.518946 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:39.525882 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.519227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40697 (local address 127.0.0.1:33624)
0925 13:16:39.519269 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.519278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.519309 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.519476 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.519477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.519588 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.519645 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.519647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.520387 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.520388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.520951 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.520953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.521385 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.521386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.521456 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.521893 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.521897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.523255 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.524811 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.524812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.524812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.524977 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.525192 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.525193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.525194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.525250 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.525505 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.525506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.525548 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.525645 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.525767 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:16:39.525882 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.519333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.98.61:40697)
0925 13:16:39.519378 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.519380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.519386 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:16:39.519409 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.519410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.519422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.519471 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.519725 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.520313 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.521027 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.521322 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.521372 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.523345 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.523347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.523347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.523354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.524728 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.525061 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.525062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.525063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.525118 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.525321 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.525322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.525322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.525440 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.525515 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.525711 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.525784 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:39.526280 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:39.526726 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.518975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.98.62:42339)
0925 13:16:39.519021 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.519022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.519187 (+ 165us) server_negotiation.cc:407] Connection header received
0925 13:16:39.519223 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.519223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.519234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.519288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.520005 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.520517 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.521231 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.521453 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.521521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.524021 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.524022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.524023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.524029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.525472 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.525905 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.525906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.525906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.525962 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.526163 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.526165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.526165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.526329 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.526505 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.526542 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.526617 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:16:39.534160 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.526542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34918)
0925 13:16:39.526842 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.526855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.526907 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.527025 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.527025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.527177 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.527209 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.527211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.528076 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.528077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.528842 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.528844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.529308 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.529308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.529347 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.529838 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.529843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.531251 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.533020 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.533021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.533021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.533274 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.533470 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.533471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.533471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.533534 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.533835 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.533836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.533887 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.533981 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.534085 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260925 13:16:39.534358 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.526573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.0.98.60:44197)
0925 13:16:39.526605 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.526607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.526958 (+ 351us) server_negotiation.cc:407] Connection header received
0925 13:16:39.526973 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.526973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.526981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.527023 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.527278 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.527997 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.528924 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.529234 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.529305 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.531346 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.531347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.531348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.531354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.532941 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.533358 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.533359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.533359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.533407 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.533598 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.533599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.533599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.533768 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.533821 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.534022 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.534142 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":15}
I20260925 13:16:39.535089 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.527458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.0.98.61:40697)
0925 13:16:39.527502 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.527503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.527546 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:16:39.527612 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.527613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.527620 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.527672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.528215 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.528903 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.529719 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.529920 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.529992 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.532259 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.532260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.532261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.532268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.533960 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.534328 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.534329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.534329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.534384 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.534577 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.534578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.534578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.534697 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.534885 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.534919 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.534993 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:16:39.536091 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: Bootstrap starting.
I20260925 13:16:39.536638 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:39.537112 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:39.537434 22816 log.cc:824] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:39.538195 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: No bootstrap required, opened a new log
I20260925 13:16:39.537887 22640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e31074e287e54cc0b2fea031c4a176b9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44142
I20260925 13:16:39.539675 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.539820 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63da2a59dc8444669133f3c128474846, State: Initialized, Role: FOLLOWER
I20260925 13:16:39.539911 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.540232 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:39.540203 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.540297 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:16:39.542078 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:39.542145 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:39.542435 22748 master_runner.cc:186] Error getting master registration for 127.0.98.62:42339: OK, instance_id { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" instance_seqno: 1790342199157088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:16:39.544505 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:16:39.562045 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } attempt: 3
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:16:39.570133 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.562538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.0.98.60:44197)
0925 13:16:39.562600 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.562603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.562615 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:16:39.562641 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.562643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.562653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.562722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.563076 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.563553 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.564746 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.564944 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.565022 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.567052 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.567054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.567055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.567063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.568761 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.569214 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.569215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.569215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.569279 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.569482 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.569483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.569483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.569720 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.569792 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.569942 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.570028 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:16:39.570125 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.562465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34930)
0925 13:16:39.562515 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.562528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.562578 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.562741 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.562742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.562946 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.563000 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.563002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.563647 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.563656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.564633 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.564635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.565018 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.565019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.565074 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.565568 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.565573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.566947 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.568889 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.568891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.568891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.569122 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.569353 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.569353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.569354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.569417 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.569783 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.569784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.569833 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.569903 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.570010 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 13:16:39.571831 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: Bootstrap starting.
I20260925 13:16:39.572611 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:39.572916 22741 log.cc:824] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:39.573815 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: No bootstrap required, opened a new log
I20260925 13:16:39.575903 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.576059 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8970b0c51814fe59f5b9981c4d0f8e7, State: Initialized, Role: FOLLOWER
I20260925 13:16:39.576160 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.576535 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:39.576493 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.576768 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:16:39.578543 22847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:39.578604 22847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:39.578826 22673 master_runner.cc:186] Error getting master registration for 127.0.98.62:42339: OK, instance_id { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" instance_seqno: 1790342199157088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:16:39.581806 22673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:16:39.602243 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } attempt: 3
Sep 25 13:16:39 dist-test-slave-2d0s krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:16:39.610754 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.602641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44197 (local address 127.0.0.1:34932)
0925 13:16:39.602702 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:39.602715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:39.602758 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:39.602936 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:39.602937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:39.603164 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:39.603220 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.603222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.604191 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.604193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.605003 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:39.605005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.605482 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.605483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:39.605549 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.606105 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:39.606110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:39.607699 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:39.609604 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.609606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.609606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.609856 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.610077 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:39.610078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:39.610078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:39.610146 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:39.610418 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:39.610419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:39.610470 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:16:39.610542 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:16:39.610644 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:16:39.610754 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.602712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.98.60:44197)
0925 13:16:39.602757 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.602759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.602772 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:16:39.602840 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.602842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.602854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.602907 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.603337 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.604070 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.605086 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.605388 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.605465 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.607783 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.607785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.607785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.607793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.609511 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.609942 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.609943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.609944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.610005 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.610216 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.610217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.610217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.610349 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.610414 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.610577 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.610656 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:16:39.612533 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9: Bootstrap starting.
I20260925 13:16:39.613281 22666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:39.613574 22666 log.cc:824] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:39.614545 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9: No bootstrap required, opened a new log
I20260925 13:16:39.615928 22666 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.616060 22666 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31074e287e54cc0b2fea031c4a176b9, State: Initialized, Role: FOLLOWER
I20260925 13:16:39.616145 22666 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.616501 22640 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 13:16:39.616446 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.616555 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:39.616564 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:39.617352 22640 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.616474 22666 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:39.617595 22640 leader_election.cc:288] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [CANDIDATE]: Term 1 election: Requested vote from peers d8970b0c51814fe59f5b9981c4d0f8e7 (127.0.98.61:40697), 63da2a59dc8444669133f3c128474846 (127.0.98.60:44197)
I20260925 13:16:39.618913 22715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e31074e287e54cc0b2fea031c4a176b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7"
I20260925 13:16:39.619071 22715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:39.619980 22715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31074e287e54cc0b2fea031c4a176b9 in term 1.
W20260925 13:16:39.620225 22865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 13:16:39.620285 22614 leader_election.cc:302] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8970b0c51814fe59f5b9981c4d0f8e7, e31074e287e54cc0b2fea031c4a176b9; no voters:
W20260925 13:16:39.620321 22865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:39.620446 22850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:39.620689 22850 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [term 1 LEADER]: Becoming Leader. State: Replica: e31074e287e54cc0b2fea031c4a176b9, State: Running, Role: LEADER
I20260925 13:16:39.620657 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e31074e287e54cc0b2fea031c4a176b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "63da2a59dc8444669133f3c128474846"
I20260925 13:16:39.620766 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:39.620781 22850 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } }
I20260925 13:16:39.621165 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31074e287e54cc0b2fea031c4a176b9. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.621246 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:39.621531 22790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31074e287e54cc0b2fea031c4a176b9 in term 1.
I20260925 13:16:39.623637 22869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:39.623843 22869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:39.634785 22790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [term 1 FOLLOWER]: Refusing update from remote peer e31074e287e54cc0b2fea031c4a176b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:39.635129 22867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:39.636605 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31074e287e54cc0b2fea031c4a176b9. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.636749 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:39.637181 22715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [term 1 FOLLOWER]: Refusing update from remote peer e31074e287e54cc0b2fea031c4a176b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:39.637444 22867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:39.638783 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31074e287e54cc0b2fea031c4a176b9. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.638840 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.622027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.0.98.62:42339)
0925 13:16:39.622069 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.622071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.622848 (+ 777us) server_negotiation.cc:407] Connection header received
0925 13:16:39.622953 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.622954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.622964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.622995 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.624557 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.625298 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.626367 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.626649 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.626691 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.628324 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.628325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.628326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.628333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.630028 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.636820 (+ 6792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.636822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.636823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.636891 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.637892 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.637893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.637894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.638046 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.638153 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.638652 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.638752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:39.638880 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:39.640231 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.640336 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:39.640825 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.640944 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:39.641178 22873 mvcc.cc:205] Tried to move back new op lower bound from 7333241649700818944 to 7333241649647353856. Current Snapshot: MvccSnapshot[applied={T|T < 7333241649700818944}]
I20260925 13:16:39.641350 22876 mvcc.cc:205] Tried to move back new op lower bound from 7333241649700818944 to 7333241649647353856. Current Snapshot: MvccSnapshot[applied={T|T < 7333241649700818944}]
I20260925 13:16:39.642210 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.642305 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:39.642764 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e31074e287e54cc0b2fea031c4a176b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31074e287e54cc0b2fea031c4a176b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } peers { permanent_uuid: "d8970b0c51814fe59f5b9981c4d0f8e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40697 } } peers { permanent_uuid: "63da2a59dc8444669133f3c128474846" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44197 } } }
I20260925 13:16:39.642871 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:39.643744 22874 mvcc.cc:205] Tried to move back new op lower bound from 7333241649700818944 to 7333241649647353856. Current Snapshot: MvccSnapshot[applied={T|T < 7333241649700818944}]
I20260925 13:16:39.646023 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.622278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.60:44197)
0925 13:16:39.622312 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.622314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.629818 (+ 7504us) server_negotiation.cc:407] Connection header received
0925 13:16:39.629924 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.629925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.629936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.629965 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.630847 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.631554 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.634768 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.635042 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.635081 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.640466 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.640468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.640469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.640476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.642424 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.643226 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.643227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.643227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.643292 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.643510 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.643511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.643512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.643644 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.643732 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.645855 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.645940 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:16:39.648046 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:39.622598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.98.61:40697)
0925 13:16:39.622639 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.622640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.630305 (+ 7665us) server_negotiation.cc:407] Connection header received
0925 13:16:39.630361 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.630363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.630375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.630431 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.633140 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.633819 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.637138 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:39.637423 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:39.637466 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:39.643936 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:39.643938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:39.643939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:39.643947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:39.645653 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.647138 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.647139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.647139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.647206 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:39.647426 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:39.647427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:39.647428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:39.647565 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:39.647642 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:39.647873 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 13:16:39.647958 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:39.649780 22869 catalog_manager.cc:1347] Generated new cluster ID: 5a2c347cbcb245a8945751fd4b43f045
I20260925 13:16:39.649857 22869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:39.677737 22869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:39.679551 22869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:39.691695 22869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e31074e287e54cc0b2fea031c4a176b9: Generated new TSK 0
I20260925 13:16:39.691926 22869 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:16:39.715318 22669 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.713542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.0.98.62:42339)
0925 13:16:39.713590 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.713592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.714608 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 13:16:39.714720 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.714722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.714733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.714765 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.715245 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44160: BlockingRecv error: recv got EOF from 127.0.0.1:44160 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:16:39.716048 22819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.713957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.98.60:44197)
0925 13:16:39.713998 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.714000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.715564 (+ 1564us) server_negotiation.cc:407] Connection header received
0925 13:16:39.715657 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.715658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.715669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.715696 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.715987 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34946: BlockingRecv error: recv got EOF from 127.0.0.1:34946 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:16:39.716722 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.714325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.98.61:40697)
0925 13:16:39.714365 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.714367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.716268 (+ 1901us) server_negotiation.cc:407] Connection header received
0925 13:16:39.716357 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.716359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.716369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.716397 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.716662 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33644: BlockingRecv error: recv got EOF from 127.0.0.1:33644 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 13:16:39.718726 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.718004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.0.98.61:40697)
0925 13:16:39.718042 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.718043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.718254 (+ 211us) server_negotiation.cc:407] Connection header received
0925 13:16:39.718363 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.718364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.718381 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.718408 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.718681 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33658: BlockingRecv error: recv got EOF from 127.0.0.1:33658 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 13:16:39.721719 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.720953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.0.98.61:40697)
0925 13:16:39.720985 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.720987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.721213 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:16:39.721311 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.721312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.721323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.721351 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.721649 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33660: BlockingRecv error: recv got EOF from 127.0.0.1:33660 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 13:16:39.725625 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.724847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.61:40697)
0925 13:16:39.724886 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.724888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.725091 (+ 203us) server_negotiation.cc:407] Connection header received
0925 13:16:39.725196 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.725197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.725206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.725234 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.725560 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33672: BlockingRecv error: recv got EOF from 127.0.0.1:33672 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 13:16:39.731649 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.730888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.0.98.61:40697)
0925 13:16:39.730929 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.730931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.731143 (+ 212us) server_negotiation.cc:407] Connection header received
0925 13:16:39.731238 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.731239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.731250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.731278 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.731594 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33682: BlockingRecv error: recv got EOF from 127.0.0.1:33682 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:16:39.741959 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.741064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.98.61:40697)
0925 13:16:39.741115 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.741125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.741248 (+ 123us) server_negotiation.cc:407] Connection header received
0925 13:16:39.741360 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.741361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.741374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.741448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.741883 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33698: BlockingRecv error: recv got EOF from 127.0.0.1:33698 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:16:39.759949 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.759109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.0.98.61:40697)
0925 13:16:39.759159 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.759161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.759297 (+ 136us) server_negotiation.cc:407] Connection header received
0925 13:16:39.759401 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.759402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.759414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.759511 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.759865 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33706: BlockingRecv error: recv got EOF from 127.0.0.1:33706 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 13:16:39.794035 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.793277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.61:40697)
0925 13:16:39.793337 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.793340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.793444 (+ 104us) server_negotiation.cc:407] Connection header received
0925 13:16:39.793479 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.793480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.793491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.793571 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.793968 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33712: BlockingRecv error: recv got EOF from 127.0.0.1:33712 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 13:16:39.860044 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.859157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.0.98.61:40697)
0925 13:16:39.859240 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.859242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.859357 (+ 115us) server_negotiation.cc:407] Connection header received
0925 13:16:39.859446 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.859447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.859458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.859527 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.859975 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33726: BlockingRecv error: recv got EOF from 127.0.0.1:33726 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 13:16:39.990025 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:39.989325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.0.98.61:40697)
0925 13:16:39.989418 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:39.989420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:39.989439 (+ 19us) server_negotiation.cc:407] Connection header received
0925 13:16:39.989510 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:39.989511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:39.989521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:39.989575 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:39.989950 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33728: BlockingRecv error: recv got EOF from 127.0.0.1:33728 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 13:16:40.247938 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:40.247278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.98.61:40697)
0925 13:16:40.247367 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:40.247370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:40.247381 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:16:40.247429 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:40.247430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:40.247444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:40.247518 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:40.247888 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33732: BlockingRecv error: recv got EOF from 127.0.0.1:33732 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 13:16:40.542610 22833 catalog_manager.cc:1259] Loaded cluster ID: 5a2c347cbcb245a8945751fd4b43f045
I20260925 13:16:40.542697 22833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: loading cluster ID for follower catalog manager: success
I20260925 13:16:40.543463 22833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: acquiring CA information for follower catalog manager: success
I20260925 13:16:40.543709 22833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63da2a59dc8444669133f3c128474846: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:16:40.579051 22847 catalog_manager.cc:1259] Loaded cluster ID: 5a2c347cbcb245a8945751fd4b43f045
I20260925 13:16:40.579131 22847 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: loading cluster ID for follower catalog manager: success
I20260925 13:16:40.579975 22847 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: acquiring CA information for follower catalog manager: success
I20260925 13:16:40.580322 22847 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8970b0c51814fe59f5b9981c4d0f8e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:16:40.763953 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:40.761377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.0.98.61:40697)
0925 13:16:40.761529 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:40.761531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:40.761541 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:16:40.761612 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:40.761613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:40.761719 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:40.761800 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:40.761982 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.762520 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.763271 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.763464 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.763518 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:40.763655 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:40.763770 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:40.763829 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:40.763863 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:16:40.763882 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260925 13:16:40.768419 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:40.766141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.0.98.60:44197)
0925 13:16:40.766258 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:40.766260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:40.766269 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:40.766343 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:40.766344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:40.766379 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:40.766446 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:40.766578 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.767059 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.767750 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.767936 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.767985 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:40.768115 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:40.768230 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:40.768287 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:40.768332 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 13:16:40.768350 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260925 13:16:40.773204 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:40.770335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.0.98.61:40697)
0925 13:16:40.770386 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:40.770388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:40.770614 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:16:40.770741 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:40.770742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:40.770753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:40.770780 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:40.770980 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.771696 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.772464 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.772676 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.772753 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:40.772903 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:40.772939 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:40.773002 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:40.773068 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 13:16:40.773092 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:16:40.773689 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:40.770386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.60:44197)
0925 13:16:40.770436 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:40.770438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:40.770606 (+ 168us) server_negotiation.cc:407] Connection header received
0925 13:16:40.770636 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:40.770637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:40.770647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:40.770701 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:40.770911 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.771416 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.772781 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.773099 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.773144 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:40.773362 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:40.773396 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:40.773428 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:40.773545 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 13:16:40.773569 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:16:40.773856 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:40.770246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.0.98.62:42339)
0925 13:16:40.770456 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:40.770458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:40.770467 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:40.770514 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:40.770515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:40.770572 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:40.770697 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:40.770788 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.771586 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.772781 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:40.773103 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:40.773346 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:40.773434 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:40.773577 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:40.773639 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:40.773732 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:16:40.773761 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":55,"threads_started":1}
I20260925 13:16:40.775673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22598
I20260925 13:16:40.781569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22673
I20260925 13:16:40.786958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22748
2026-09-25T13:16:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2911 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:16:40 dist-test-slave-2d0s krb5kdc[22962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:16:40 dist-test-slave-2d0s krb5kdc[22962](info): set up 2 sockets
Sep 25 13:16:40 dist-test-slave-2d0s krb5kdc[22962](info): commencing operation
krb5kdc: starting...
W20260925 13:16:41.817986 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/test-user.keytab.
Sep 25 13:16:41 dist-test-slave-2d0s krb5kdc[22962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:16:41.938004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43149,127.0.98.61:35873,127.0.98.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:16:42.025549 22975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:16:42.025933 22975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:42.025974 22975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:42.026018 22975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:42.027362 22975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:42.027436 22975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:42.027462 22975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:42.027494 22975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:42.027523 22975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:42.029255 22975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43149,127.0.98.61:35873,127.0.98.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:42.029561 22975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:42.029839 22975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:42.034736 22981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:42.034723 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:42.034757 22983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:42.035019 22975 server_base.cc:1034] running on GCE node
I20260925 13:16:42.035264 22975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:42.035597 22975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:42.036780 22975 hybrid_clock.cc:648] HybridClock initialized: now 1790342202036765 us; error 44 us; skew 500 ppm
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:42.049736 22975 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:16:42.050244 22975 webserver.cc:460] Webserver started at http://127.0.98.62:39937/ using document root <none> and password file <none>
I20260925 13:16:42.050490 22975 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:42.050560 22975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:42.050720 22975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:42.051929 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data/instance:
uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "22e4f15a585389aad2bf1b016707e4ce"
server_key_iv: "c179eb55449cb35e3c3e591660498a7d"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.052330 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal/instance:
uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "22e4f15a585389aad2bf1b016707e4ce"
server_key_iv: "c179eb55449cb35e3c3e591660498a7d"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.053987 22975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 13:16:42.054872 22990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:42.055133 22975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:42.055253 22975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal
uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "22e4f15a585389aad2bf1b016707e4ce"
server_key_iv: "c179eb55449cb35e3c3e591660498a7d"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.055335 22975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:42.079324 22975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:42.081913 22975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:42.082087 22975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:42.086228 22975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43149
I20260925 13:16:42.086297 23042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43149 every 8 connection(s)
I20260925 13:16:42.086640 22975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:16:42.087703 23043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:42.089380 23043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.093642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 22975
I20260925 13:16:42.093784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-0/wal/instance
I20260925 13:16:42.093971 392 external_mini_cluster.cc:1428] Setting key 08cedb707279a380f895312b4d2dcee4
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:42.100749 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.090141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43149 (local address 127.0.0.1:58732)
0925 13:16:42.090325 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.090368 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.090837 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.091110 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.091112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.091374 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.091551 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.091555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.092601 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.092603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.093534 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.093538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.094303 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.094304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.094389 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.095001 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.095012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.097072 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.099101 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.099108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.099111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.099335 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.099611 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.099612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.099612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.099705 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.100094 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.100100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.100170 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.100276 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.100524 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":90,"thread_start_us":79,"threads_started":1}
I20260925 13:16:42.100751 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.090184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.0.98.62:43149)
0925 13:16:42.090385 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.090390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.090439 (+ 49us) server_negotiation.cc:407] Connection header received
0925 13:16:42.090972 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.090974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.090993 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.091095 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.091645 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.092479 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.093636 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.094171 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.094295 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.097193 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.097202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.097202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.097210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.098996 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.099433 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.099437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.099437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.099533 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.099769 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.099771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.099771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.100015 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.100107 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.100301 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.100456 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":111,"threads_started":1}
I20260925 13:16:42.101725 23043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260925 13:16:42.102134 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.102018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37636)
0925 13:16:42.102064 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.102111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 13:16:42.102305 22991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
I20260925 13:16:42.102766 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.102677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37638)
0925 13:16:42.102722 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.102748 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:16:42.102957 23043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 13:16:42.120903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35873
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43149,127.0.98.61:35873,127.0.98.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:16:42.152230 23043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } attempt: 1
I20260925 13:16:42.152803 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.152665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37654)
0925 13:16:42.152730 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.152772 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 13:16:42.153155 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.153044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37662)
0925 13:16:42.153091 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.153115 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 13:16:42.153246 23043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
W20260925 13:16:42.205312 23050 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:16:42.205616 23050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:42.205708 23050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:42.205775 23050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:42.207096 23050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:42.207167 23050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:42.207192 23050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:42.207233 23050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:42.207265 23050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:42.208961 23050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43149,127.0.98.61:35873,127.0.98.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:42.209273 23050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:42.209517 23050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:16:42.214386 23058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:42.214485 23055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:42.214504 23056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:42.214835 23050 server_base.cc:1034] running on GCE node
I20260925 13:16:42.215124 23050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:42.215498 23050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:42.216661 23050 hybrid_clock.cc:648] HybridClock initialized: now 1790342202216643 us; error 34 us; skew 500 ppm
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:42.219439 23050 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 13:16:42.219987 23050 webserver.cc:460] Webserver started at http://127.0.98.61:33789/ using document root <none> and password file <none>
I20260925 13:16:42.220245 23050 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:42.220291 23050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:42.220433 23050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:42.221397 23043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } attempt: 2
I20260925 13:16:42.221617 23050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data/instance:
uuid: "d796acf998e34bb99c16a223fe7eac54"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "882423f556324d8cdcdc82e4b3a27637"
server_key_iv: "f7ca9dfadc97f2b19bed1694512c3c83"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.221887 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.221759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37664)
0925 13:16:42.221803 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.221838 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 13:16:42.222103 23050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal/instance:
uuid: "d796acf998e34bb99c16a223fe7eac54"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "882423f556324d8cdcdc82e4b3a27637"
server_key_iv: "f7ca9dfadc97f2b19bed1694512c3c83"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.222254 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.222150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37678)
0925 13:16:42.222204 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.222235 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 13:16:42.222355 23043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35873: connect: Connection refused (error 111)
I20260925 13:16:42.223642 23050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:16:42.224494 23065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:42.224756 23050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:42.224851 23050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal
uuid: "d796acf998e34bb99c16a223fe7eac54"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "882423f556324d8cdcdc82e4b3a27637"
server_key_iv: "f7ca9dfadc97f2b19bed1694512c3c83"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.224911 23050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:42.231599 23050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:42.233731 23050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:42.233853 23050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:42.237773 23050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35873
I20260925 13:16:42.237836 23117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35873 every 8 connection(s)
I20260925 13:16:42.238178 23050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:16:42.239238 23118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:42.240935 23118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.246142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 23050
I20260925 13:16:42.246271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-1/wal/instance
I20260925 13:16:42.246479 392 external_mini_cluster.cc:1428] Setting key a20e09df7c1867a6f6f6a8ce99885c1d
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:42.251224 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.241722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.0.98.62:43149)
0925 13:16:42.241770 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.241773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.242020 (+ 247us) server_negotiation.cc:407] Connection header received
0925 13:16:42.242555 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.242556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.242573 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.242645 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.243335 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.243856 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.245029 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.245248 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.245338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.247831 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.247834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.247834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.247844 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.249743 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.250163 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.250164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.250165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.250232 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.250530 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.250532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.250532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.250661 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.250735 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.251032 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.251130 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:16:42.251415 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.241692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43149 (local address 127.0.0.1:58746)
0925 13:16:42.241890 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.241927 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.242441 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.242721 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.242724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.243010 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.243220 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.243224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.243958 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.243960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.244832 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.244834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.245372 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.245373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.245518 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.246162 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.246172 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.247701 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.249832 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.249839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.249841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.250076 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.250316 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.250317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.250317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.250421 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.250727 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.250734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.250880 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.250989 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.251169 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260925 13:16:42.252067 23118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:16:42.261194 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.252339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37682)
0925 13:16:42.252386 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.252398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.252440 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.252722 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.252723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.252909 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.252997 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.253000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.253947 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.253949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.254845 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.254848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.255390 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.255391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.255457 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.255961 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.255966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.258145 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.260121 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.260122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.260122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.260290 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.260506 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.260507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.260508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.260561 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.260850 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.260851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.260889 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.260961 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.261072 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:16:42.261221 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.252422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.0.98.61:35873)
0925 13:16:42.252573 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.252578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.252595 (+ 17us) server_negotiation.cc:407] Connection header received
0925 13:16:42.252614 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.252616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.252632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.252689 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.253083 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.253865 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.254951 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.255310 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.255374 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.258236 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.258249 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.258250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.258258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.260013 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.260379 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.260382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.260383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.260437 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.260623 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.260624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.260625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.260784 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.260876 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.260998 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.261127 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260925 13:16:42.261847 23118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.262219 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.262102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53784)
0925 13:16:42.262147 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.262191 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:16:42.262343 23067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
I20260925 13:16:42.262740 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.262648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53800)
0925 13:16:42.262695 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.262718 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:16:42.262843 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 13:16:42.271951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOnpMs/build/release/bin/kudu
/tmp/dist-test-taskpOnpMs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46467
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43149,127.0.98.61:35873,127.0.98.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:16:42.312084 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } attempt: 1
I20260925 13:16:42.312676 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.312526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53812)
0925 13:16:42.312594 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.312646 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:16:42.313026 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.312951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53818)
0925 13:16:42.312987 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.313010 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 13:16:42.313124 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
I20260925 13:16:42.313500 23043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } attempt: 3
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:16:42.321090 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.313847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37684)
0925 13:16:42.313905 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.313917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.313969 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.314101 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.314102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.314289 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.314366 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.314369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.315044 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.315045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.315923 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.315926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.316313 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.316314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.316396 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.316855 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.316860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.318354 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.320065 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.320066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.320066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.320245 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.320464 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.320465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.320466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.320531 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.320815 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.320816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.320861 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.320920 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.321003 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:16:42.321127 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.313906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.98.61:35873)
0925 13:16:42.313948 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.313950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.313966 (+ 16us) server_negotiation.cc:407] Connection header received
0925 13:16:42.314034 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.314035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.314044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.314086 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.314459 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.314960 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.316018 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.316241 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.316295 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.318438 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.318440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.318441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.318449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.319950 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.320315 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.320316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.320316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.320379 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.320598 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.320599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.320599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.320737 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.320802 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.320958 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.321038 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:16:42.321697 23043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.322075 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.321970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53834)
0925 13:16:42.322017 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.322054 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 13:16:42.322386 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.322312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53844)
0925 13:16:42.322351 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.322372 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 13:16:42.322480 23043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
I20260925 13:16:42.353688 23043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } attempt: 1
I20260925 13:16:42.354324 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.354114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53860)
0925 13:16:42.354228 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.354286 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 13:16:42.355032 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.354905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53876)
0925 13:16:42.354957 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.354997 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:16:42.355170 23043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
W20260925 13:16:42.359185 23125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:16:42.359462 23125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:16:42.359485 23125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:16:42.359519 23125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:16:42.360744 23125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:16:42.360785 23125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:16:42.360800 23125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:16:42.360811 23125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:16:42.360842 23125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:16:42.362469 23125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43149,127.0.98.61:35873,127.0.98.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:16:42.362759 23125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:16:42.363014 23125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:16:42.368260 23125 server_base.cc:1034] running on GCE node
W20260925 13:16:42.368243 23133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:42.368242 23131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:16:42.368243 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:16:42.368803 23125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:16:42.369122 23125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:16:42.370302 23125 hybrid_clock.cc:648] HybridClock initialized: now 1790342202370283 us; error 37 us; skew 500 ppm
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:16:42.372942 23125 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 13:16:42.373426 23125 webserver.cc:460] Webserver started at http://127.0.98.60:38389/ using document root <none> and password file <none>
I20260925 13:16:42.373720 23125 fs_manager.cc:359] Metadata directory not provided
I20260925 13:16:42.373770 23125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:16:42.373914 23125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:16:42.375036 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data/instance:
uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "ffc3cda61cd10d1b0148b4da2a4f7e93"
server_key_iv: "3c0ae92f88e46e1e7ec4cbb889a414e7"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.375439 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal/instance:
uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "ffc3cda61cd10d1b0148b4da2a4f7e93"
server_key_iv: "3c0ae92f88e46e1e7ec4cbb889a414e7"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.377014 23125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:16:42.377919 23140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:16:42.378157 23125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:16:42.378250 23125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal
uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8"
format_stamp: "Formatted at 2026-09-25 13:16:42 on dist-test-slave-2d0s"
server_key: "ffc3cda61cd10d1b0148b4da2a4f7e93"
server_key_iv: "3c0ae92f88e46e1e7ec4cbb889a414e7"
server_key_version: "encryptionkey@0"
I20260925 13:16:42.378314 23125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:16:42.381284 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } attempt: 2
I20260925 13:16:42.381775 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.381616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53880)
0925 13:16:42.381701 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.381740 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 13:16:42.382123 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.382015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53890)
0925 13:16:42.382062 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.382091 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:16:42.382218 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46467: connect: Connection refused (error 111)
I20260925 13:16:42.395133 23125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:16:42.397275 23125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:16:42.397447 23125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:16:42.402007 23125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46467
I20260925 13:16:42.402040 23192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46467 every 8 connection(s)
I20260925 13:16:42.402429 23125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:16:42.403484 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:16:42.405107 23193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.407516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOnpMs/build/release/bin/kudu as pid 23125
I20260925 13:16:42.407603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOnpMs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790341932215442-392-0/minicluster-data/master-2/wal/instance
I20260925 13:16:42.407835 392 external_mini_cluster.cc:1428] Setting key d5e9e78c36fb27312b629ef0006554b9
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:16:42.415534 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.405898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.0.98.62:43149)
0925 13:16:42.405960 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.405962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.406157 (+ 195us) server_negotiation.cc:407] Connection header received
0925 13:16:42.406711 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.406712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.406725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.406777 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.407481 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.408091 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.409634 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.409888 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.409961 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.412488 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.412490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.412490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.412498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.414090 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.414513 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.414514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.414515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.414577 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.414841 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.414842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.414843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.414969 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.415038 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.415343 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.415429 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:16:42.415753 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.405857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43149 (local address 127.0.0.1:58760)
0925 13:16:42.406024 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.406068 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.406600 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.406858 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.406862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.407139 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.407378 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.407384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.408179 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.408181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.409552 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.409555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.409998 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.410000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.410127 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.410683 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.410694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.412393 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.414178 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.414188 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.414190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.414435 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.414654 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.414655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.414655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.414772 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.415039 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.415047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.415205 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.415311 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.415503 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260925 13:16:42.416329 23193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.417500 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.408219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.0.98.60:46467)
0925 13:16:42.408408 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.408412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.408511 (+ 99us) server_negotiation.cc:407] Connection header received
0925 13:16:42.408552 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.408554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.408571 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.408640 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.409419 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.410217 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.411238 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.411533 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.411595 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.414391 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.414400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.414401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.414408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.416235 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.416639 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.416643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.416643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.416721 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.416916 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.416918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.416918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.417068 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.417255 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.417301 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.417387 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:16:42.424291 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.416693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.98.61:35873)
0925 13:16:42.416745 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.416748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.416756 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:16:42.416780 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.416781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.416790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.416841 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.417100 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.417710 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.418527 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.418885 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.418970 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.421395 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.421396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.421397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.421406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.423071 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.423459 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.423461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.423461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.423527 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.423744 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.423746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.423746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.423873 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.423953 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.424106 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.424185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:16:42.424291 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.416621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35873 (local address 127.0.0.1:37700)
0925 13:16:42.416666 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.416676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.416709 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.416847 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.416847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.416988 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.417025 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.417026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.417793 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.417794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.418453 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.418455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.418989 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.418990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.419045 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.419513 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.419519 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.421244 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.423194 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.423196 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.423197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.423381 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.423605 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.423606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.423606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.423657 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.423943 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.423944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.423995 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.424078 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.424185 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 13:16:42.424789 23193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:16:42.426709 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.418365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.0.98.62:43149)
0925 13:16:42.418401 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.418403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.418413 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:16:42.418435 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.418436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.418445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.418506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.419118 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.419942 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.420671 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.421017 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.421088 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.423668 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.423669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.423669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.423677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.425419 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.425831 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.425832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.425833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.425896 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.426088 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.426089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.426089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.426217 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.426417 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.426454 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.426532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:16:42.430305 23043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } attempt: 2
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:16:42.432330 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.425075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53920)
0925 13:16:42.425173 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.425184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.425228 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.425357 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.425357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.425474 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.425510 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.425511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.426219 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.426220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.426703 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.426704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.427167 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.427168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.427250 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.427658 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.427664 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.429151 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.431011 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.431012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.431013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.431236 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.431441 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.431441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.431442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.431507 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.432041 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.432042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.432082 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.432143 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.432236 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 13:16:42.432667 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.425136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.0.98.60:46467)
0925 13:16:42.425178 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.425180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.425221 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:16:42.425269 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.425270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.425283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.425355 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.425584 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.426131 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.426794 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.427099 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.427166 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.429245 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.429247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.429247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.429255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.430930 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.431313 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.431315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.431315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.431371 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.431578 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.431579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.431580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.431721 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.432016 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.432521 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.432592 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:16:42.436187 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.427328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.0.98.61:35873)
0925 13:16:42.427365 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.427369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.427398 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:16:42.427422 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.427422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.427432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.427483 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.428080 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.428877 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.429808 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.430030 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.430103 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.432819 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.432821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.432821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.432829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.434887 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.435330 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.435331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.435332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.435388 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.435603 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.435604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.435604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.435766 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.435944 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.436017 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.436099 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:16:42.437284 23193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: Bootstrap starting.
I20260925 13:16:42.437826 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:16:42.438206 23193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:42.438552 23193 log.cc:824] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:42.439473 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.430605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53936)
0925 13:16:42.430658 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.430669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.430706 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.431060 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.431061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.431236 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.431287 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.431289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.432444 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.432445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.433611 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.433615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.434430 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.434432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.434487 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.434952 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.434957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.436475 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.438349 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.438350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.438350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.438527 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.438767 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.438768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.438768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.438829 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.439121 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.439122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.439173 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.439254 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.439367 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:16:42.439507 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.430644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.0.98.60:46467)
0925 13:16:42.430899 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.430901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.430908 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:16:42.430928 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.430928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.430939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.431483 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.431528 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.432349 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.433971 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.434332 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.434640 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.436550 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.436552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.436552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.436559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.438233 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.438607 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.438608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.438608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.438685 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.438905 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.438906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.438907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.439050 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.439131 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.439306 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.439382 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":184,"threads_started":1}
I20260925 13:16:42.439152 23017 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58772
I20260925 13:16:42.440639 23193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: No bootstrap required, opened a new log
I20260925 13:16:42.441067 23043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3: Bootstrap starting.
I20260925 13:16:42.441834 23043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:42.442073 23043 log.cc:824] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:42.442588 23193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.442739 23193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b0c890ef5e4d0e88c2eb0c85537ad8, State: Initialized, Role: FOLLOWER
I20260925 13:16:42.442781 23043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3: No bootstrap required, opened a new log
I20260925 13:16:42.442832 23193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.443091 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.443172 23193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:42.443254 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:42.444003 23043 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.444159 23043 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48ba4a9e9c1406e95d5ab81da95c0b3, State: Initialized, Role: FOLLOWER
I20260925 13:16:42.444249 23043 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.444566 23043 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:42.444590 23017 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 13:16:42.444638 23017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:42.444530 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.444878 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:16:42.445508 23213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:42.445557 23213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:42.445554 23017 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.446372 23167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8"
I20260925 13:16:42.446477 23167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [term 0 FOLLOWER]: Advancing to term 1
W20260925 13:16:42.446708 23225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:16:42.446753 23225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:42.447476 23167 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48ba4a9e9c1406e95d5ab81da95c0b3 in term 1.
I20260925 13:16:42.447721 22992 leader_election.cc:302] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b0c890ef5e4d0e88c2eb0c85537ad8, f48ba4a9e9c1406e95d5ab81da95c0b3; no voters:
I20260925 13:16:42.447834 23210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:16:42.447968 23210 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [term 1 LEADER]: Becoming Leader. State: Replica: f48ba4a9e9c1406e95d5ab81da95c0b3, State: Running, Role: LEADER
I20260925 13:16:42.448047 23210 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.448457 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48ba4a9e9c1406e95d5ab81da95c0b3. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.445410 23125 master_runner.cc:186] Error getting master registration for 127.0.98.62:43149: OK, instance_id { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" instance_seqno: 1790342202085004 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:16:42.448531 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:42.448575 23125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:16:42.447243 22975 master_runner.cc:186] Error getting master registration for 127.0.98.61:35873: OK, instance_id { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" instance_seqno: 1790342202236656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:16:42.450022 22975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:16:42.445896 23017 leader_election.cc:288] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [CANDIDATE]: Term 1 election: Requested vote from peers d796acf998e34bb99c16a223fe7eac54 (127.0.98.61:35873), 47b0c890ef5e4d0e88c2eb0c85537ad8 (127.0.98.60:46467)
I20260925 13:16:42.461184 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.451290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.0.98.62:43149)
0925 13:16:42.451397 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.451399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.451689 (+ 290us) server_negotiation.cc:407] Connection header received
0925 13:16:42.451786 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.451787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.451798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.451830 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.452559 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.453297 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.454448 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.454747 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.454791 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.456886 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.456887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.456887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.456896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.458617 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.459510 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.459513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.459513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.459664 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.460252 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.460254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.460254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.460433 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.460541 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.460959 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.461065 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 13:16:42.461491 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.451492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.0.98.61:35873)
0925 13:16:42.451545 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.451547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.451556 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:16:42.451609 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.451610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.451621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.451676 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.452536 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.453296 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.454448 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.454784 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.454822 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.457229 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.457230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.457231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.457240 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.459029 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.459760 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.459763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.459764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.459928 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.460245 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.460246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.460247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.460433 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.461036 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.461276 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.461373 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:16:42.463266 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.451440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.98.60:46467)
0925 13:16:42.451496 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.451498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.451876 (+ 378us) server_negotiation.cc:407] Connection header received
0925 13:16:42.452039 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.452040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.452051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.452083 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.453384 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.454213 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.456129 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.456473 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.456518 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.459796 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.459798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.459798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.459806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.461571 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.462029 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.462031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.462031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.462091 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.462504 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.462505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.462505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.462754 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.462891 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.463040 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.463154 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:16:42.467996 23229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:16:42.468638 23229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:16:42.473435 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } attempt: 3
I20260925 13:16:42.474012 23167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [term 1 FOLLOWER]: Refusing update from remote peer f48ba4a9e9c1406e95d5ab81da95c0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:42.474375 23210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:42.476063 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48ba4a9e9c1406e95d5ab81da95c0b3. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.476183 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 13:16:42 dist-test-slave-2d0s krb5kdc[22962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:16:42.482477 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.473906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46467 (local address 127.0.0.1:53954)
0925 13:16:42.473957 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:16:42.473970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:16:42.474014 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:16:42.474167 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:16:42.474168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:16:42.474381 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:16:42.474440 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.474442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.475362 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.475363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.476197 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:16:42.476199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.476690 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.476691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:16:42.476756 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.477305 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:16:42.477311 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:16:42.478971 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:16:42.481258 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.481259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.481260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.481482 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.481757 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:16:42.481758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:16:42.481759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:16:42.481819 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:16:42.482133 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:16:42.482133 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:16:42.482184 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:16:42.482247 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:16:42.482362 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:16:42.482781 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:42.473941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.0.98.60:46467)
0925 13:16:42.473993 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.473995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.474003 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:16:42.474100 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.474101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.474112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.474163 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.474553 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.475255 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.476292 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:42.476568 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:42.476630 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:42.479077 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:16:42.479078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:16:42.479079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:16:42.479118 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:16:42.481165 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.481572 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.481573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.481573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.481685 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:16:42.481883 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:16:42.481884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:16:42.481884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:16:42.482029 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:16:42.482107 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:42.482622 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 13:16:42.482702 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:16:42.484303 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: Bootstrap starting.
I20260925 13:16:42.485126 23118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: Neither blocks nor log segments found. Creating new log.
I20260925 13:16:42.485491 23118 log.cc:824] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: Log is configured to *not* fsync() on all Append() calls
I20260925 13:16:42.486356 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: No bootstrap required, opened a new log
I20260925 13:16:42.488505 23118 raft_consensus.cc:348] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.488662 23118 raft_consensus.cc:729] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d796acf998e34bb99c16a223fe7eac54, State: Initialized, Role: FOLLOWER
I20260925 13:16:42.488773 23118 consensus_queue.cc:260] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } }
I20260925 13:16:42.489130 23118 sys_catalog.cc:564] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:16:42.489185 23092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d796acf998e34bb99c16a223fe7eac54"
I20260925 13:16:42.489274 23092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:16:42.490329 23092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48ba4a9e9c1406e95d5ab81da95c0b3 in term 1.
I20260925 13:16:42.490672 23091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [term 1 FOLLOWER]: Refusing update from remote peer f48ba4a9e9c1406e95d5ab81da95c0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:16:42.490960 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.491014 23210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:16:42.491088 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:42.491171 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.491240 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:42.491354 23237 mvcc.cc:205] Tried to move back new op lower bound from 7333241661329813504 to 7333241661227941888. Current Snapshot: MvccSnapshot[applied={T|T < 7333241661329813504}]
I20260925 13:16:42.491550 23229 catalog_manager.cc:1347] Generated new cluster ID: 976920c6218b4ca0a5f2530432f76cdb
I20260925 13:16:42.491611 23229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:16:42.494133 23238 mvcc.cc:205] Tried to move back new op lower bound from 7333241661329813504 to 7333241661227941888. Current Snapshot: MvccSnapshot[applied={T|T < 7333241661329813504}]
I20260925 13:16:42.494316 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.494405 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:42.498000 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48ba4a9e9c1406e95d5ab81da95c0b3. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.498106 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:42.498327 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.498437 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:16:42.499585 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48ba4a9e9c1406e95d5ab81da95c0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } peers { permanent_uuid: "d796acf998e34bb99c16a223fe7eac54" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35873 } } peers { permanent_uuid: "47b0c890ef5e4d0e88c2eb0c85537ad8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46467 } } }
I20260925 13:16:42.499692 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:16:42.499725 23252 catalog_manager.cc:1259] Loaded cluster ID: 976920c6218b4ca0a5f2530432f76cdb
I20260925 13:16:42.499773 23252 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: loading cluster ID for follower catalog manager: success
W20260925 13:16:42.499861 23252 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 13:16:42.499894 23252 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:16:42.500918 23229 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:16:42.501472 23229 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:16:42.506853 23229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48ba4a9e9c1406e95d5ab81da95c0b3: Generated new TSK 0
I20260925 13:16:42.507100 23229 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:16:42.533296 23046 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.531833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.0.98.62:43149)
0925 13:16:42.531885 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.531888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.532558 (+ 670us) server_negotiation.cc:407] Connection header received
0925 13:16:42.532673 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.532674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.532686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.532718 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.533213 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58798: BlockingRecv error: recv got EOF from 127.0.0.1:58798 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 13:16:42.534098 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.532255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.0.98.60:46467)
0925 13:16:42.532297 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.532299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.533555 (+ 1256us) server_negotiation.cc:407] Connection header received
0925 13:16:42.533655 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.533656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.533688 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.533720 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.534031 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53958: BlockingRecv error: recv got EOF from 127.0.0.1:53958 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:16:42.535095 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.534439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.98.61:35873)
0925 13:16:42.534482 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.534485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.534613 (+ 128us) server_negotiation.cc:407] Connection header received
0925 13:16:42.534703 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.534704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.534714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.534744 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.535034 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37714: BlockingRecv error: recv got EOF from 127.0.0.1:37714 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:16:42.537647 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.536855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.0.98.61:35873)
0925 13:16:42.536891 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.536893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.537128 (+ 235us) server_negotiation.cc:407] Connection header received
0925 13:16:42.537228 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.537229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.537248 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.537274 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.537604 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37718: BlockingRecv error: recv got EOF from 127.0.0.1:37718 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 13:16:42.540686 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.539937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.0.98.61:35873)
0925 13:16:42.539971 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.539973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.540182 (+ 209us) server_negotiation.cc:407] Connection header received
0925 13:16:42.540276 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.540277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.540287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.540316 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.540638 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37734: BlockingRecv error: recv got EOF from 127.0.0.1:37734 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 13:16:42.544777 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.543936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.98.61:35873)
0925 13:16:42.543987 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.543989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.544233 (+ 244us) server_negotiation.cc:407] Connection header received
0925 13:16:42.544330 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.544331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.544342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.544372 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.544707 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37740: BlockingRecv error: recv got EOF from 127.0.0.1:37740 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 13:16:42.551044 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.550167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.0.98.61:35873)
0925 13:16:42.550210 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.550212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.550452 (+ 240us) server_negotiation.cc:407] Connection header received
0925 13:16:42.550571 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.550572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.550585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.550617 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.550986 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37752: BlockingRecv error: recv got EOF from 127.0.0.1:37752 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:16:42.561384 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.560449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.0.98.61:35873)
0925 13:16:42.560495 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.560511 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.560773 (+ 262us) server_negotiation.cc:407] Connection header received
0925 13:16:42.560881 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.560882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.560896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.560927 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.561319 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37768: BlockingRecv error: recv got EOF from 127.0.0.1:37768 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:16:42.580220 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.579178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.0.98.61:35873)
0925 13:16:42.579229 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.579232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.579532 (+ 300us) server_negotiation.cc:407] Connection header received
0925 13:16:42.579654 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.579655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.579669 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.579703 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.580152 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37782: BlockingRecv error: recv got EOF from 127.0.0.1:37782 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:16:42.614799 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.613803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.0.98.61:35873)
0925 13:16:42.613852 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.613855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.614130 (+ 275us) server_negotiation.cc:407] Connection header received
0925 13:16:42.614244 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.614245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.614258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.614289 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.614735 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37796: BlockingRecv error: recv got EOF from 127.0.0.1:37796 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:16:42.681152 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.680352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.98.61:35873)
0925 13:16:42.680438 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.680441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.680594 (+ 153us) server_negotiation.cc:407] Connection header received
0925 13:16:42.680680 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.680681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.680694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.680751 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.681090 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37798: BlockingRecv error: recv got EOF from 127.0.0.1:37798 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 13:16:42.811280 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:42.810601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.0.98.61:35873)
0925 13:16:42.810665 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:42.810667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:42.810762 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:16:42.810831 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:42.810832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:42.810844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:42.810888 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:42.811200 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37806: BlockingRecv error: recv got EOF from 127.0.0.1:37806 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 13:16:43.069386 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:16:43.068660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.0.98.61:35873)
0925 13:16:43.068744 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:43.068746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:43.068757 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:16:43.068835 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:43.068836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:43.068848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:43.068916 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:16:43.069313 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37816: BlockingRecv error: recv got EOF from 127.0.0.1:37816 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 13:16:43.446031 23213 catalog_manager.cc:1259] Loaded cluster ID: 976920c6218b4ca0a5f2530432f76cdb
I20260925 13:16:43.446127 23213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: loading cluster ID for follower catalog manager: success
I20260925 13:16:43.446938 23213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: acquiring CA information for follower catalog manager: success
I20260925 13:16:43.447242 23213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47b0c890ef5e4d0e88c2eb0c85537ad8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:16:43.501016 23252 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: acquiring CA information for follower catalog manager: success
I20260925 13:16:43.501385 23252 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d796acf998e34bb99c16a223fe7eac54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:16:43.585399 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:43.582572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.98.61:35873)
0925 13:16:43.582722 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:43.582724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:43.582757 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:16:43.582911 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:43.582912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:43.582973 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:43.583078 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:43.583226 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.583733 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.584553 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.584756 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.584839 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:43.585037 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:43.585150 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:43.585239 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:43.585266 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:16:43.585288 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 13:16:43.589634 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:43.587181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.0.98.60:46467)
0925 13:16:43.587340 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:43.587342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:43.587350 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:16:43.587389 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:43.587390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:43.587426 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:43.587497 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:43.587593 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.588151 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.588865 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.589051 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.589135 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:43.589303 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:43.589404 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:43.589462 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:43.589495 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 13:16:43.589518 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260925 13:16:43.594106 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:43.591420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.60:46467)
0925 13:16:43.591456 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:43.591458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:43.591573 (+ 115us) server_negotiation.cc:407] Connection header received
0925 13:16:43.591672 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:43.591674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:43.591683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:43.591725 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:43.591831 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.592580 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.593437 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.593697 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.593758 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:43.593871 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:43.593902 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:43.593946 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:43.594001 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 13:16:43.594024 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:16:43.594501 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:43.591414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.0.98.61:35873)
0925 13:16:43.591458 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:43.591460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:43.591536 (+ 76us) server_negotiation.cc:407] Connection header received
0925 13:16:43.591633 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:43.591634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:43.591647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:43.591676 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:43.591868 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.592589 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.593646 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.593971 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.594042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:43.594228 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:43.594264 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:43.594293 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:43.594397 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 13:16:43.594420 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:16:43.594831 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:16:43.591317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.98.62:43149)
0925 13:16:43.591489 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 13:16:43.591492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:16:43.591554 (+ 62us) server_negotiation.cc:407] Connection header received
0925 13:16:43.591634 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:16:43.591635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:16:43.591688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:16:43.591766 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:16:43.591923 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.592396 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.593496 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:16:43.593823 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:16:43.593881 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:16:43.594201 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:16:43.594328 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:16:43.594383 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:16:43.594713 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 13:16:43.594743 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
I20260925 13:16:43.596427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 22975
I20260925 13:16:43.601715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 23050
I20260925 13:16:43.606588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOnpMs/build/release/bin/kudu with pid 23125
2026-09-25T13:16:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2818 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5730 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271396 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST